acceptodds
Under review as a conference paper at ICLR 2027

SMART: Zero-Shot Sim-to-Real Articulated Object Manipulation via Large-Scale Synthetic Pretraining

Abstract

Interacting with articulated objects is essential for intelligent embodiments, but real-world large-scale demonstration collection for this remains challenging due to the precise contact and constraint-following motions involved. Although simulation provides a promising alternative, existing synthetic data efforts cover limited articulated-object categories, while general-purpose synthesis pipelines lack explicit designs for part-level semantics and articulation constraints, hindering agentic task generation and scalable synthesis of high-quality articulated-manipulation demonstrations. To bridge this gap, we introduce SMART, a scalable system leveraging large-scale Synthesized Manipulation demonstrations for ARTiculated-object manipulation. At its core, we develop SMART-Sim, a simulation platform with articulation-aware design that enables effective task generation and efficient demonstration collection. Building on SMART-Sim, we apply agentic task generation and design a scalable distributed synthesis system, using them to synthesize SMART-Data, comprising over 1M demonstrations across 44 atomic task types, 5 robot setups, and 2,507 articulated objects. The VLA model pretrained on SMART-Data shows competitive performance on simulation benchmarks and achieves zero-shot sim-to-real transfer and scalable performance in real-world articulated-object manipulation tasks. This highlights the potential of synthetic demonstrations in providing effective and scalable supervision for improving VLA model performance in contact-rich articulated-object manipulation.

Primary area:
datasets and benchmarks
Keywords:
Articulated Object Manipulation, Robot Simulation, Large-Scale Data Synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_87rWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents SMART, a framework for generating large-scale synthetic demonstrations for articulated-object manipulation. It combines articulation-aware asset annotation, constraint-aware motion generation, LLM-based task generation, and distributed data synthesis to construct over one million demonstrations. Experiments on simulation benchmarks and three real-world robot platforms show improved downstream performance, data efficiency, and promising zero-shot sim-to-real transfer without real-world demonstrations.

Critical strengths
  1. Large-scale synthetic data generation for articulated-object manipulation. SMART integrates articulation-aware motion generation, task composition, and distributed data synthesis to produce over one million demonstrations. This provides a potentially valuable data source for VLA pretraining.

  2. Promising real-world sim-to-real results. Experiments across three robot platforms and 12 tasks demonstrate that policies trained using synthetic demonstrations can achieve high real-world performance without real-world demonstrations.

Critical weaknesses
  1. Insufficiently established novelty over existing approaches. The main contributions appear to lie in scaling articulated-object assets and integrating articulation-aware motion generation into an existing synthetic data pipeline. Given the overlap with prior work in agentic task generation and synthetic manipulation data, the technical advance over existing approaches remains unclear.
  2. Insufficient experimental evidence for the claimed advantages of SMART-Data. The paper lacks controlled comparisons with representative synthetic pretraining approaches, making it difficult to determine whether SMART-Data provides unique benefits beyond those of large-scale synthetic pretraining in general. The modest improvement on LIBERO and the absence of a real-data-pretrained baseline on RoboCasa365 further limit the evidence supporting the dataset's effectiveness and generalizability.
Questions for authors
  1. Limited novelty over existing synthetic data generation pipelines. SMART combines articulation-aware asset annotation, constraint-aware motion generation, and agentic task composition. However, asset preparation, annotation, and the 23 manipulation skills still require substantial prior engineering, while agentic task generation and skill composition have been explored in prior work, such as GenManip and HumanoidGen. The paper would benefit from a clearer characterization of its technical advances over existing simulation pipelines and a quantitative analysis of the human intervention required in practice.
  2. Insufficient evidence of unique task coverage and dataset advantages. Many tasks shown in the demonstration video, such as opening cabinets and manipulating microwave doors and knobs, appear to overlap with InternData-A1. The paper does not clearly establish which task types or manipulation capabilities are uniquely enabled by SMART.
  3. Limited evidence for the effectiveness of SMART-Data pretraining. On LIBERO, SMART-VLA improves over training from scratch by only 2.1 percentage points and remains 1.0 point below the real-data-pretrained baseline. Such gains may reflect the general benefits of pretraining rather than the unique value of SMART-Data. On RoboCasa365, the improvement is larger, but the corresponding real-data-pretrained baseline is missing. Also, the paper lacks a direct comparison with existing synthetic pretraining datasets.
  4. Limited insight into real-world sim-to-real transfer. Although SMART-VLA achieves promising real-world results, the paper provides limited analysis of why it outperforms the baselines. The contributions of articulation-aware motion generation, contact quality, and domain randomization remain unclear, making it difficult to understand the underlying mechanisms and the conditions under which the proposed approach is most effective. A systematic failure analysis would provide stronger evidence for the proposed method's benefits.
5 comments
AuthorsOpenReview ↗

Rebuttal and Response to Reviewer 87rW

We appreciate your detailed and constructive feedback. We understand your concerns regarding the novelty and contributions of our work, and we address them below.

Addressing the concerns in the weakness section

We demonstrate the novelty and effectiveness of SMART by elaborating on its design and presenting comparative experimental analyses as follows:

1. The novelty and effectiveness of SMART’s design.

SMART’s articulation-aware designs are twofold:

(1) Part-level annotations and an articulation-aware skill library that let an agent generate executable articulated-object manipulation tasks. The designs in this part include:

  • Part-level manipulation annotation
  • Action frames per part
  • Articulation-aware skill library
  • Annotation-driven task generation procedure

(2) A demonstration generation step designed for articulated-object manipulation trajectories. The designs in this part include:

  • Constraint-compliant end-effector trajectories
  • Sustained contact throughout in-contact manipulation
  • Diversification of trajectories and part-level visual appearance

The effectiveness of these designs is supported by our results on synthesis efficiency (task-generation automation and trajectory success rate) and on downstream performance (comparison with InternData-A1 and ablations, which is shown in the section below). Please see the main response for the detailed design description, results, and conclusions.

2. The comparison experiments and ablation studies.

Since multiple reviewers raised concerns regarding this section, we address it in our main response at the top of this page. (Please search for “additional experiments” on this page). We refer the reviewer to that section for details on the experimental designs.

The results are shown in the table below:

Data SourceDrawer pulling (%)Microwave Knob Rotating (%)Cabinet Door Sliding (%)Toolbox Lid Flipping (%)Avg (%)
SMART-Data-1000h66.773.366.773.370.0
InternData-A1-Art-1000h73.360.053.346.758.3
SMART-Data-1000h (No-Part-MTL-Rand)53.353.353.366.756.7
SMART-Data-1000h (No-ART-Design)66.760.060.053.360.0

We summarize our observations and analysis as follows:

  • Articulated-object manipulation demonstrations synthesized by SMART yield better training performance than those from InternData-A1 on downstream tasks. This is evidenced by SMART-Data-1000h’s higher average task success rate (70.0 vs 58.3).
    • We observe a substantial performance gap on Microwave Knob Rotating, Cabinet Door Sliding, and Toolbox Lid Flipping. These tasks demand persistent in-contact manipulation and strict constraint adherence.
  • The articulated-object manipulation designs in SMART improve the training performance of the synthesized dataset. Disabling either part-level material randomization or the articulation-aware design degrades model performance (70.0 vs 56.7 vs 60.0).

3. Failure case analysis

We refer the reviewer to our main response for experiment details.

We report the results in the table below.

Data SourceMiss the target part (%)Lose contact when manipulating (%)Constraint Violation (%)Others (%)
SMART-Data-1000h11.76.76.75.0
InternData-A1-Art-1000h13.311.713.33.3
SMART-Data-1000h (No-Part-MTL-Rand)18.310.010.05.0
SMART-Data-1000h (No-ART-Design)11.715.011.71.7

We summarize our observations and analysis as follows:

  • The advantage of SMART-Data over InternData-A1-Art-1000h lies in constraint adherence and contact maintenance.
    • It halves the constraint-violation rate (6.7% vs 13.3%) and reduces the loss-of-contact rate by approximately 40% (6.7% vs 11.7%), while the miss-the-target-part rate remains nearly comparable (11.7% vs 13.3%).
    • The performance gain therefore originates from the constraint- and contact-aware designs. Since the per-task performance gap follows the same trend as that of the ablated variants, we defer the detailed analysis to the ablation and failure case analysis below.
  • Part-level material randomization mainly benefits perception, namely the visual grounding of the target manipulable part: disabling this component increases the miss-the-target-part rate from 11.7% to 18.3%.
    • In the drawer-pulling task, the handle and drawer body share identical material and color, yielding weak visual cues for identifying the manipulable part.
    • The policy trained on data augmented with part-level material randomization exhibits more stable performance under such visual ambiguity, demonstrating the effectiveness of this design for visual grounding.
  • Articulation-aware designs mainly facilitate action inference: removing them doubles the loss-of-contact rate (6.7% to 15.0%) and elevates the constraint-violation rate to 11.7%.
    • For example, in the cabinet door sliding task, policies trained without these designs often lose grip on the handle and only partially open the door, or violate the rail constraint by pulling the door outward and damaging it.
    • When opening a toolbox lid, the policy pushes the entire box away rather than rotating the lid along the hinge constraint, so it fails to open the box.
    • These observations demonstrate the contribution of articulation-aware designs to inferring actions that respect joint kinematic constraints.

We hope the experiments above can address your concerns in this part.

AuthorsOpenReview ↗

Answering the questions

Answering the questions

Q1: Regarding your concerns about the novelty of our work, we have elaborated on our designs and provided additional comparative experiments and ablation studies above. We further discuss this point below:

  • The mentioned works, namely GenManip and HumanoidGen, are relevant to agentic / LLM-driven task generation yet have different focuses compared to ours.
  • GenManip is a simulation platform for evaluating policy generalization. It proposes task-oriented scene graphs to represent task semantic layouts and enable effective LLM-driven task generation.
    • Its task generation pipeline centers on task schema generation and does not produce the action sequences required for articulated object manipulation.
    • While it annotates feasible states of articulated objects, it lacks the part-level visual and manipulation annotations that constitute a core contribution of our work.
    • It collects trajectories via teleoperation and leverages MimicGen for articulated object manipulation. By contrast, SMART investigates how to synthesize trajectories for articulated objects, extend the pipeline to more object categories, and scale up data collection efficiently.
  • HumanoidGen targets the adaptation of LLM-driven task generation for dexterous dual-arm manipulation.
    • It does not perform part-level annotation for articulated objects and adopts only a generic manipulation formulation.
    • It validates its method on a restricted set of articulated object categories and does not pursue data scaling.

In summary, for articulated object manipulation tasks, none of these prior works incorporate designs for agent-driven task generation or articulation-aware motion generation.

Prior datasets such as InternData-A1 serve as suitable baselines for comparison, yet they do not investigate how to effectively extend demonstration synthesis to a broader range of articulated objects or how to scale up data collection efficiently.

We highlight this research gap in our introduction and related work sections, which motivates the contributions of SMART.

Q2:

Q2.1 The comparison target mentioned above, InternData-A1, states that it uses 14 categories of articulated objects in its dataset. SMART uses 23 categories of articulated objects for demonstration synthesis, as shown in Fig. 4(c).

InternData-A1 provides 15 articulated object manipulation tasks, whereas SMART provides 44 types of atomic articulated object manipulation tasks.

This provides direct evidence of SMART’s advantage over InternData-A1 in articulated object category coverage and task coverage.

Q2.2 We show the advantage of SMART-Data in our response to the weakness section above.

Q3: We understand your concern about the pretraining effect, and clarify the role of the two simulation benchmarks as follows.

  • LIBERO and RoboCasa365 are designed to evaluate general manipulation capability. They cover a broad mix of tasks, and articulated-object manipulation is only one part of them.
    • SMART focuses on scaling data synthesis and supporting sim-to-real transfer for articulated-object manipulation, and SMART-Data consists mainly of such tasks.
    • We therefore use these two benchmarks to evaluate how SMART-Data pretraining affects the general manipulation capability of the policy. We do not claim, based on the results on these benchmarks, that SMART-Data has unique value for pretraining.
  • The concrete value of SMART-Data is demonstrated by our real-world evaluations and by the additional experiments reported in our response to the weakness section.
    • These tasks are contact-rich articulated-object manipulation tasks, which align with the capabilities that SMART-Data targets. Training on SMART-Data yields better model performance on these tasks, which directly demonstrates the unique value of SMART-Data.
    • We further explore the source of this value in our additional experiments and ablation studies. We refer the reviewer to the main response and to our response to the weakness section for details.

Q4: We have elaborated on our designs and provided additional experiments and analysis in our response to the weaknesses section.


We hope the above responses address the concerns you raised regarding our work.

We remain open to further discussion, and we would be grateful if you would consider raising your score should these concerns be resolved.

Reviewer_FKX1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds a simulation system especially for articulated objects. They provide large-scale articulated-object data and propose a vision-language-action model to pretrain on this dataset. Their results demonstrate the benefits of pretraining on their proposed dataset and show that these benefits can successfully transfer to real-world deployment and beat existing general action models like π0.5\pi_{0.5}.

Critical strengths
  1. The paper proposes a large-scale simulated robot demonstration dataset with a simulator, which improves upon previous methods in terms of skills, tasks, and articulated-object trajectories.
  2. The proposed model, SMART-VLA, demonstrates strong performance gains on real robots, with the success rate increasing from 50.0% for the model without first-stage pretraining to 76.7%.
Critical weaknesses
  1. The contribution of the dataset is not clear enough. I do not see a clear ablation study testing whether task diversity, the number of demonstrations, or the model architecture contributes to the performance gains from pretraining. Although the authors have shown scaling from 1 to 500 episodes for fine-tuning on downstream tasks, the gains from the pretraining stage need to be further explained to demonstrate the strengths of the proposed dataset and simulator. They also do not clearly distinguish their large-scale simulation dataset from those of other similar methods.
Questions for authors
  1. The paper stresses articulated-object manipulation in its simulator as a critical component, so I would like to see an ablation study in which the authors manually relax the articulation constraints and reduce part-level randomization to examine how performance changes.
1 comment
AuthorsOpenReview ↗

Rebuttal and Response to Reviewer FKX1

We appreciate your helpful review feedback.

We see that your main concern lies in the contribution of our dataset and the lack of ablations. We therefore provide the following experiments and address your concerns in the weakness and question sections together as follows:

1. The comparison experiments and ablation studies.

Since multiple reviewers raised concerns regarding this section, we address it in our main response at the top of this page. (Please search for “additional experiments” on this page). We refer the reviewer to that section for details on the experiment designs.

The results are shown in the table below (we follow the same evaluation protocol as the main experiment):

Data SourceDrawer pulling (%)Microwave Knob Rotating (%)Cabinet Door Sliding (%)Toolbox Lid Flipping (%)Avg (%)
SMART-Data-1000h66.773.366.773.370.0
InternData-A1-Art-1000h73.360.053.346.758.3
SMART-Data-1000h (No-Part-MTL-Rand)53.353.353.366.756.7
SMART-Data-1000h (No-ART-Design)66.760.060.053.360.0

We summarize our observations and analysis as follows:

  • Articulated-object manipulation demonstrations synthesized by SMART yield better training performance than those from InternData-A1 on downstream tasks. This is evidenced by SMART-Data-1000h’s advantage in average task success rate (70.0 vs 58.3).
    • We observe a substantial performance gap on Microwave Knob Rotating, Cabinet Door Sliding, and Toolbox Lid Flipping. These tasks demand persistent in-contact manipulation and strict constraint adherence.
  • The articulated-object manipulation designs in SMART benefit the training performance of the synthesized dataset. Disabling either part-level material randomization or the articulation-aware design degrades model performance (70.0 vs 56.7 vs 60.0).

2. Failure case analysis

We provide experiment details in the main response and report the results in the table below.

Data SourceMiss the target part (%)Lose contact when manipulating (%)Constraint Violation (%)Others (%)
SMART-Data-1000h11.76.76.75.0
InternData-A1-Art-1000h13.311.713.33.3
SMART-Data-1000h (No-Part-MTL-Rand)18.310.010.05.0
SMART-Data-1000h (No-ART-Design)11.715.011.71.7

We summarize our observations and analysis as follows:

  • The advantage of SMART-Data over InternData-A1-Art-1000h lies in constraint adherence and contact maintenance.
    • It halves the constraint-violation rate (6.7% vs 13.3%) and reduces the loss-of-contact rate by approximately 40% (6.7% vs 11.7%).
    • The performance gain therefore originates from the constraint- and contact-aware designs. We defer the detailed analysis to the ablation and failure case analysis below.
  • Part-level material randomization mainly benefits perception, namely the visual grounding of the target manipulable part.
    • Disabling this component increases the miss-the-target-part rate from 11.7% to 18.3%.
    • In the drawer-pulling task, the handle and the drawer body share identical material and color, resulting in weak visual cues for identifying the manipulable part.
    • The policy trained on data augmented with part-level material randomization exhibits more stable performance under such visual ambiguity, demonstrating the effectiveness of this design for visual grounding.
  • Articulation-aware designs mainly facilitate action inference: removing them doubles the loss-of-contact rate (6.7% to 15.0%) and elevates the constraint-violation rate to 11.7%.
    • For example, in the cabinet door sliding task, policies trained without these designs often lose grip on the handle and only partially open the door, or violate the rail constraint by pulling the door outward and damaging it.
    • These observations demonstrate the contribution of articulation-aware designs to inferring actions that respect joint kinematic constraints.

We hope these experimental results address your core concerns, and we remain open to further discussion regarding our work. We would greatly appreciate it if you could consider raising your score should our responses resolve your concerns.

Reviewer_VSaKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SMART, a scalable pipeline for generating large-scale synthetic articulated object manipulation demonstration data. It has three components: 1> SMART-sim: an object articulation-aware simulation plattform for automatic asset annotation, motion generation, skill design and domain randomization, 2> an agentic LLM-based task generation approach, and 3> a distributed data collection system with a scheduler plus separate collection, rendering, and conversion workers, that leads to SMART-Data containing over 1M demonstrations across five robot setups and 44 atomic task types. The key finding is that availability of large-scale synthetic data (SMART-Data), can enable pretraining of VLAs improving their down-stream performance and supports sim-to-real transfer wiithout collecting real-world robot demonstrations for the target tasks.

Critical strengths
  1. I think this paper tackles a genuinely useful problem. Doing articulated manipulation needs pretty precise interaction with handles, joints, and other functional parts (as they need sustained contacts throughout the operation). A pipeline built around that makes sense, and gathering this scale of data is a strong contribution, I believe. Overall, the whole system seems well thought out. The part annotations connect to the skill definitions and constrained motion generation, and the collection pipeline filters unsuccessful trajectories before spending resources on rendering. I think these components fit together well. The contribution is mainly in the system and data, and I think that is reasonable for this work.

2> The comparison with real-data pretraining is a good for understanding the contribution. SMART-Data pretraining gives better results on the real-world articulated tasks, with 76.7% success compared to 69.4% with real data pretraining, and 50.0% without the first-stage pretraining. this gives useful evidence that synthetic data can be a good alternative to real-world data for pretraining, especially for articulated manipulation, which is important as it is extremely difficult to scale real-world data. Additionally, evaluating on three robot platforms also gives morre convincing evidence than demonstrating the approach on one setup. And finally, the downstream data-scaling experiments are also useful and show that pretrained model could be valuable when only a small amount of task-specific synthetic data is available. There is evidence of benefit beyond the specific real-world tasks. On RoboCasa365, the gains over the model without pretraining are more noticeable on composite tasks, suggesting that the pretraining also helps with tasks involving multiple steps (Table 1).

Critical weaknesses

1> The meaning of zeroshot transfer is not very clear in the main paper. Appendix D.1 says that images of the target objects are used to generate simulation assets, which are then processed into articulated objects, also, the policies also receive task-specific synthetic post-training. while it is a valid setting for zero-shot sim-to-real transfer, it does not establish generalization to unseen object instances without additional preparation (manual or automatic). I think it would be good tomake this distinction more explicit and explain how much information about each real object is needed. A small evaluation on additional instances without rebuilding their assets would strengthen the result. Also, in terms of the pipeline, it is not clear which parts are actually done by the VLM and which are manual (annotation, calibration, identifying manipulable parts). The paper acknowledges substantial asset post-processing and allows human refinement of generated tasks. It would help to show the fraction of task configurations that work without correction, typical preparation time per asset, and generation success rates for different task lengths. This is important as an automatic pipeline should require considerably less effort for extending to more objects.

2> The efficacy of pretraining has been the primary focus of the evaluations, but how do the design choices help the result is not clear. it is difficult to tell how much comes from articulation-aware generation, part-level randomization, task diversity, or simply having more manipulation data. An ablation on a few of these choices would help. InternData-A1 seems to be the closest comparison here (Tabl 2), especially since it also includes articulated-object manipulation. A controlled pretraining comparison would help show the benefit of SMART-Data. The authors could pretrain the same backbone on the two datasets with similar training budgets, and then keep the downstream training the same. Tables 2 and 5 compare dataset and framework details, but it is still not clear which dataset leads to better transfer.

Questions for authors

Clarification requests:

1> I would like to see ablations of a few design choices (e.g., articulation-aware generation or part-level randomization), keeping data volume fixed to clarify their contributions beyond simply using more data.

2> I would like to see the comparison for pretraining on SMART-Data and InternData-A1 using the same backbone, similar training budgets, and identical downstream training to see which gives better transfer?

3> How much manual work is needed to prepare a target object for the real-world experiments? How are its joint parameters and their bounds, and physical properties obtained?

4> What overlap exists between SMART-Data pretraining assets and the task-specific assets used for real-world evaluation?

5> What are the most common real-world failures: reaching the wrong contact point, losing contact, or failing to follow the joint motion?

6> How would the action annotations work for multi-finger hands (the figures show only parallel grippers) or any other type which is not in annotation? The simulator lists an hand gripper, but it is not used in the dataset or experiments. Would the annotations need any changes?

7> Is there randomization in terms of the grasp location also for articulated-objects? For e.g. a microwave has a long handle to open it, but the robot could grasp it at any location on the handle.

suggestions:

1> It would be good to add scaling experiment for all the 12 tasks used for evaluating sim-to-real. This would strengthen the claim.

2> Also, it would be good to increase the number of tasks for a stronger evaluation.

5 comments
AuthorsOpenReview ↗

Rebuttal and Response to Reviewer VSaK

We thank reviewer VSaK for the helpful review feedback and constructive suggestions to improve the paper. We clarify the reviewer’s confusion and address the concerns as follows:

Addressing the concerns in the weakness section

W1. We address your concerns as follows:

W1.1: We understand your concerns about the possible ambiguity in the meaning of zero-shot sim-to-real transfer. We clarify this part as follows:

  • First, we explain the meaning of zero-shot sim-to-real policy transfer.
    • Zero-shot sim-to-real transfer is a specific problem setting, which means achieving successful policy transfer from simulation to the real world without using real-world demonstration data for domain adaptation.
    • We distinguish it from zero-shot generalization, which means generalizing to unseen tasks or objects without task-specific training.
  • This expression is used in peer-reviewed work.
    • InternData-A1 uses the same expression in its abstract: “The resulting model also exhibits surprisingly zero-shot sim-to-real transfer on several challenging tasks.”
    • Other works such as DemoStart and RE3SIM also use the same expression and problem setting, achieving zero-shot sim-to-real transfer with only simulation demonstrations.
    • We follow the same problem setting as these previous works.
  • We have taken measures to avoid ambiguity in our paper.
    • Wherever we mention zero-shot sim-to-real transfer, we explicitly write “without any real-world demonstrations” to make the setting unambiguous.
    • Our experiments are also designed to support the zero-shot sim-to-real transfer problem setting rather than zero-shot generalization. We do not claim any contribution to zero-shot generalization on unseen objects.

We will further revise our expressions in the manuscript and add additional explanation sections in the appendix to help readers understand the problem setting.

W1.2: About the details of the asset annotation pipeline (before the step that locates the action frames):

  • We first load the articulated object files and extract their articulation skeletons. The extracted skeleton may lack part semantics, but it provides an initial estimate of joint types and ranges, possibly with errors. We acknowledge that omitting this prestep from the main paper may cause confusion.
  • We then apply the render-and-identify step described in the paper. A VLM is used to identify the manipulable parts from the rendered images.
  • We then use the VLM to further calibrate and annotate the articulation and part semantics accordingly. The parts identified by the VLM complement the part semantics missing from the articulation skeleton and help correct joint types and ranges that are clearly wrong. The result is a complete articulation structure with full part semantics, ready for the subsequent action-frame locating step.

We hope the explanation above addresses your concerns, and we will revise the corresponding parts of our manuscript.

W1.3: We’d like to share the following typical time budgets for constructing a task:

StepTime Consumption
Asset Preparation (per asset, no parallel worker applied)2 min
Task Generation (if with revision)<5min
Trajectory Generation (1 worker)~10min
Rendering and Conversion (1 worker)~200min

The results above were obtained with a single GPU worker. We would like to highlight that SMART is highly scalable owing to its distributed scheduling design. Its production throughput scales linearly with the number of workers. With 8 workers, we can obtain 500 episodes in about 25 min, which is a notably high throughput.

W1.4 We also provide the ratio of automatically generated tasks (without correction) in SMART-Data in the table below.

AtomicCompositeLong-horizon
Automatically generated ratio~81%100%~40%

We observe that most atomic tasks are generated automatically and that composite tasks are assembled from validated atomic tasks without human intervention. Long-horizon tasks account for a smaller proportion, since longer horizons incur a higher failure rate.

The average success rate of the trajectories generated in stage-1 data synthesis is shown below. We would like to highlight that, given the throughput of our scalable data synthesis design, this success rate is sufficient to keep data synthesis efficient.

AtomicComposite & Long-horizon
SMART35%14%

W2: Your concerns are addressed by the experiments and analysis below:

1. The comparison experiments and ablation studies.

Since multiple reviewers raised concerns regarding this section, we address it in our main response at the top of this page. (Please search for “additional experiments” on this page). We refer the reviewer to that section for details on the experiment designs.

The results are shown in the table below (we follow the same evaluation protocol as the main experiment):

Data SourceDrawer pulling (%)Microwave Knob Rotating (%)Cabinet Door Sliding (%)Toolbox Lid Flipping (%)Avg (%)
SMART-Data-1000h66.773.366.773.370.0
InternData-A1-Art-1000h73.360.053.346.758.3
SMART-Data-1000h (No-Part-MTL-Rand)53.353.353.366.756.7
SMART-Data-1000h (No-ART-Design)66.760.060.053.360.0

We summarize our observations and analysis as follows:

  • Articulated-object manipulation demonstrations synthesized by SMART provide a better training effect compared to those from InternData-A1 in downstream tasks. This is shown by SMART-Data-1000h’s advantage in the average task success rate (70.0 vs 58.3).
    • We observed a significant performance gap in Microwave Knob Rotating, Cabinet Door Sliding, and Toolbox Lid Flipping. These tasks require persistent in-contact manipulation and strict constraint following.
  • The articulated-object manipulation designs in SMART contribute to the training effect of the synthesized dataset. Deactivating either part-level material randomization or articulation-aware design leads to a reduction in model performance (70.0 vs 56.7 vs 60.0).

We further explore the reasons for the above observations with failure case analysis.

2. Failure case analysis

We refer the reviewer to our main response for experiment setting details. We report the results in the table below.

Data SourceMiss the target part (%)Lose contact when manipulating (%)Constraint Violation (%)Others (%)
SMART-Data-1000h11.76.76.75.0
InternData-A1-Art-1000h13.311.713.33.3
SMART-Data-1000h (No-Part-MTL-Rand)18.310.010.05.0
SMART-Data-1000h (No-ART-Design)11.715.011.71.7

We summarize our observations and analysis as follows:

  • The advantage of SMART-Data over InternData-A1-Art-1000h lies in constraint following and contact maintenance.
    • It halves the constraint-violation rate (6.7% vs 13.3%) and reduces the loss-of-contact rate by about 40% (6.7% vs 11.7%), while the miss-the-target-part rate is nearly equal (11.7% vs 13.3%).
    • The performance gain therefore originates from the constraint- and contact-aware designs. Since the per-task gap follows the same pattern as that of the ablated variants, we leave the detailed analysis to the ablation and failure case analysis below.
  • Part-level material randomization mainly contributes to perception, i.e., the visual grounding of the target manipulable part: removing it raises the miss-the-target-part rate from 11.7% to 18.3%.
    • In the drawer-pulling task, the handle and the drawer body share the same material and color, which gives only a weak cue about the manipulable part.
    • The policy trained on data with part-level material randomization shows a more stable performance under this visual ambiguity, which reflects the design’s effect on visual grounding.
  • Articulation-aware designs mainly contribute to action inference: removing them doubles the loss-of-contact rate (6.7% to 15.0%) and raises the constraint-violation rate to 11.7%.
    • For example, in the cabinet door sliding task, the policy trained without these designs often loses grip on the handle, leaving the door only partially opened. In some cases, it fails to follow the rail constraint and pulls the door out, breaking the object.
    • When opening a toolbox lid, it pushes the box away instead of rotating the lid following the hinge constraint, failing to open the box.
    • These phenomena show the contribution of articulation-aware designs in inferring actions that respect the joint constraints.

We hope our responses above can address your concerns in this part.

AuthorsOpenReview ↗

Answering the questions

Answering the questions

We answer the questions as follows:

Q1: We show the ablation results in our response to W2.

Q2: We show the comparison with InternData-A1 in our response to W2.

Q3 : Q3.1: Task preparation budgets are listed in our response to W1.3. Most of the task preparation is automated, covering asset annotation, task configuration, and articulated-object reconstruction. For asset reconstruction, our generate-and-articulate pipeline needs only occasional manual correction of part segmentation, while the articulate-and-generate pipeline automatically constructs and validates articulated assets from a single reference image and a text prompt, with no manual part segmentation or joint configuration (see the “others” section of our main response). In summary, manual work in preparation is minor.

Q3.2: As described in our reply to W1.2, we obtain the articulation structures from the object assets and use the VLM to annotate and calibrate them based on the part identification results. We also provide pipelines for reconstructing articulated objects for the sim2real experiments mentioned above, and these pipelines likewise provide joint information.

We would like to emphasize that our goal is not to precisely replicate the physical properties of real-world objects but to preserve task-relevant geometry and articulation constraints. Since SMART-Sim applies part-level physical randomization and perturbs joint stiffness and damping during data synthesis, the reconstructed assets serve as simulation proxies for generating diverse and physically feasible manipulation demonstrations.

Q4: All the asset instances used for real-world evaluation are strictly excluded from the stage-1 data synthesis step.

Q5: Thank you for bringing up this question. We have answered this question in our response to the concerns of Weakness 2.

Q6. The humanoid with dexterous hands is one of the embodiments supported by SMART-Sim (Unitree H1-2 in Tab. 3) and is not used in SMART-Data or in our experiments.

SMART-Sim currently supports dexterous-hand manipulation through predefined hand gestures: we annotate action frames on the dexterous hand and pair them with predefined gestures, so the existing skills and motion generation can be reused. This simplification is proposed by HumanoidGen and can be applied to only a limited range of part geometries.

Two extensions are possible with our annotations, which we leave as future work:

  • Dexterous grasp generation. Existing dexterous grasp generation methods, optionally with VLM-based filtering, can be integrated, with our part and action-frame annotations guiding the grasp toward the functional part.
  • RL skills. For more complex dexterous manipulation, our annotations can provide pre-grasp states and the target motion of the part as a reference for training RL policies, which can then serve as skills in the same pipeline.

Q7: Yes. We apply randomization after obtaining the anchor frame from the annotation pipeline. A single manipulable part can in fact yield multiple anchor frames, which we use as randomization candidates to cover the manipulable part over its touchable surface.


We sincerely appreciate your informative feedback and constructive suggestions.

We hope our responses resolve the concerns you raised in the weakness section, and we welcome any further discussion. We would greatly appreciate it if you could consider raising your score should our responses resolve your concerns.

Reviewer_1Pv7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SMART is a synthetic-data pipeline for articulated-object manipulation. SMART-Sim combines asset and part processing, task construction, articulation-aware motion generation, and rendering. The resulting dataset contains over one million demonstrations, which are used to pretrain SMART-VLA before task-specific training. The evaluation covers LIBERO, RoboCasa365, and twelve real-robot task-platform settings across three platforms. Average real-world success is 76.7%, compared with 50.0% without synthetic pretraining. Deployment uses reconstructed target assets and task-specific synthetic demonstrations, without target-task real-robot demonstration training.

Critical strengths
  1. The data-generation system addresses practical difficulties in articulated manipulation. It uses part annotations and joint structure to generate contact motions, filters trajectories, and separates motion generation from rendering. Producing this type of data at the reported scale is a substantial engineering contribution, and the implementation details make the dataset potentially useful to other researchers.
  2. The comparison with SMART-VLA without stage-one pretraining shows a clear benefit from the full pretraining procedure. The 26.7-point average real-world improvement in Figure 5 is substantial, and gains appear across all three platforms. The deployment-height test and downstream demonstration-budget experiments provide further evidence that the initialization helps beyond a single training configuration.
Critical weaknesses
  1. The experiments do not establish which parts of the proposed synthesis method are responsible for the gains. Removing stage one changes both data exposure and training compute. The real-data-pretrained baseline is useful, but does not replace a comparison against generic synthetic pretraining under a matched budget. There is also no downstream ablation of articulation-aware contact generation, randomization, or task diversity. The scaling curves vary task-specific training data, rather than the million-episode pretraining corpus. I therefore find the evidence for useful synthetic pretraining stronger than the evidence for the particular generation method proposed here. This distinction is central to my recommendation.
  2. The real-world transfer results rely on substantial target-specific preparation. Appendix D.1 reconstructs objects from real images, manually processes their parts and articulation, and supplies 500 synthetic demonstrations for the final training stage. This qualifies as sim-to-real transfer without real demonstrations, but does not show transfer to unseen instances without rebuilding assets and retraining. Appendix D.3 also reports only 15 trials per task setting, with no uncertainty across training runs in the main comparison. The height-shift test is useful, but does not resolve the limited object coverage or establish that the smaller advantage over real-data pretraining is reliable.
Questions for authors
  1. Is there a compute-matched generic synthetic-pretraining comparison, or an ablation showing the downstream contribution of articulation-aware motion generation? How does performance change with the amount of stage-one data?
  2. How much manual work is required for each real evaluation target? Were real rollout outcomes used to tune simulation parameters or select checkpoints, and how are target assets separated from the pretraining asset pool?
  3. Are the real-world gains consistent across training seeds? Is there evidence of transfer to held-out object instances without additional asset reconstruction or task-specific training?
3 comments
AuthorsOpenReview ↗

Rebuttal and Response to Reviewer 1Pv7

We appreciate Reviewer 1Pv7 for the detailed and helpful review feedback.

We address your concerns in the weaknesses section below.

Addressing the concerns in the weaknesses section

W1: We address the concerns by providing comparison experiments as follows:

1. The comparison experiments and ablation studies.

Since multiple reviewers raised concerns regarding this section, we address it in our main response at the top of this page. (Please search for “additional experiments” on this page). We refer the reviewer to that section for details in the experimental designs.

The results are shown in the table below:

Data SourceDrawer pulling (%)Microwave Knob Rotating (%)Cabinet Door Sliding (%)Toolbox Lid Flipping (%)Avg (%)
SMART-Data-1000h66.773.366.773.370.0
InternData-A1-Art-1000h73.360.053.346.758.3
SMART-Data-1000h (No-Part-MTL-Rand)53.353.353.366.756.7
SMART-Data-1000h (No-ART-Design)66.760.060.053.360.0

We summarize our observations and analysis as follows:

  • Articulated-object manipulation demonstrations synthesized by SMART yield better training performance than those from InternData-A1 on downstream tasks. This is evidenced by SMART-Data-1000h’s higher average task success rate (70.0 vs 58.3).
    • We observe a substantial performance gap on Microwave Knob Rotating, Cabinet Door Sliding, and Toolbox Lid Flipping. These tasks demand persistent in-contact manipulation and strict constraint adherence.
  • The articulated-object manipulation designs in SMART improve the training performance of the synthesized dataset. Disabling either part-level material randomization or the articulation-aware design degrades model performance (70.0 vs 56.7 vs 60.0).

2. Failure case analysis

We refer the reviewer to our main response for experiment setting details.

We report the results in the table below.

Data SourceMiss the target part (%)Lose contact when manipulating (%)Constraint Violation (%)Others (%)
SMART-Data-1000h11.76.76.75.0
InternData-A1-Art-1000h13.311.713.33.3
SMART-Data-1000h (No-Part-MTL-Rand)18.310.010.05.0
SMART-Data-1000h (No-ART-Design)11.715.011.71.7

We summarize our observations and analysis as follows:

  • The advantage of SMART-Data over InternData-A1-Art-1000h lies in constraint adherence and contact maintenance.
    • It halves the constraint-violation rate (6.7% vs 13.3%) and reduces the loss-of-contact rate by approximately 40% (6.7% vs 11.7%), while the miss-the-target-part rate remains nearly comparable (11.7% vs 13.3%).
    • The performance gain therefore originates from the constraint- and contact-aware designs. Since the per-task performance gap follows the same trend as that of the ablated variants, we defer the detailed analysis to the ablation and failure case analysis below.
  • Part-level material randomization mainly benefits perception, namely the visual grounding of the target manipulable part: disabling this component increases the miss-the-target-part rate from 11.7% to 18.3%.
    • In the drawer-pulling task, the handle and drawer body share identical material and color, yielding weak visual cues for identifying the manipulable part.
    • The policy trained on data augmented with part-level material randomization exhibits more stable performance under such visual ambiguity, demonstrating the effectiveness of this design for visual grounding.
  • Articulation-aware designs mainly facilitate action inference: removing them doubles the loss-of-contact rate (6.7% to 15.0%) and elevates the constraint-violation rate to 11.7%.
    • For example, in the cabinet door sliding task, policies trained without these designs often lose grip on the handle and only partially open the door, or violate the rail constraint by pulling the door outward and damaging it.
    • When opening a toolbox lid, the policy pushes the entire box away rather than rotating the lid along the hinge constraint, so it fails to open the box.
    • These observations demonstrate the contribution of articulation-aware designs to inferring actions that respect joint kinematic constraints.

We hope our response above can address your concerns.

W2: We address your concerns as follows:

W2.1. Regarding your concerns about the manual effort involved in preparing for real-world transfer, we provide the following explanation:

  • For asset reconstruction, a single reference image provides the visual information needed to reconstruct task-relevant geometry and articulation structures.
    • In the generate-and-articulate pipeline, we first generate the complete mesh and then apply part segmentation and articulation skeleton reconstruction. In the other, articulate-and-generate pipeline, we first generate the articulation structure of the object and then generate the parts through agentic CAD-style code generation. The first pipeline relies on human effort for part-segmentation refinement, whereas the second pipeline is fully automated and requires only minor human prompting.
  • The agentic task generation pipeline is highly automated, and its high task generation success rate is shown below.
  • Collecting 500 episodes is well affordable given the 144 h/day data synthesis efficiency reported in the data synthesis section of our paper.

Taken together, the human effort required for sim2real preparation is minor, and the data synthesis is sufficiently efficient and affordable.

AtomicCompositeLong-horizon
Automatically generated ratio~81%100%~40%

W2.2. About the problem of zero-shot sim2real transfer:

  • First, we explain the meaning of zero-shot sim-to-real policy transfer.
    • Zero-shot sim-to-real transfer is a specific problem setting, which means achieving successful policy transfer from simulation to the real world without using real-world demonstration data for domain adaptation.
    • We distinguish it from zero-shot generalization, which means generalizing to unseen tasks or objects without task-specific training.
  • This expression is used in the peer-reviewed work.
    • InternData-A1 uses the same expression in its abstract: “The resulting model also exhibits surprisingly zero-shot sim-to-real transfer on several challenging tasks.”
    • Other works such as DemoStart and RE3SIM also use the same expression and problem setting, achieving zero-shot sim-to-real transfer with only simulation demonstrations.
    • We follow the same problem setting as these previous works.
  • We have taken measures to avoid ambiguity in our paper.
    • Wherever we mention zero-shot sim-to-real transfer, we explicitly write “without any real-world demonstrations” to make the setting unambiguous.
    • Our experiments are also designed to support the zero-shot sim-to-real transfer problem setting rather than zero-shot generalization. We do not claim any contribution to zero-shot generalization on unseen objects.

We understand that this concept can be confusing, and we believe our clarification above can address it. We will further revise our expressions in the manuscript and add additional explanation sections in the appendix to help readers understand the problem setting.

W2.3 We conduct four additional experimental rounds and report standard deviations for our main experiments. The results remain stable with no statistically significant differences. We will include this update in our revised manuscript.

AuthorsOpenReview ↗

Answering the questions

Answering the questions

Q1:

Q1.1 We have shown our additional experiments and analysis in our response to Weakness 1.

Q1.2 We conducted our stage-1 scaling experiment with three training configurations: no SMART-Data, 1000 h of SMART-Data, and the full SMART-Data. We summarize the results in the table below:

Data Volume0 hour1000 hours~4600 hours
Avg. Success Rate in AC1 tasks50.070.085.0

Providing additional SMART-Data continues to improve training performance, though gains diminish as the dataset grows.

  • The first 1000 hours lift the average success rate from 50.0% to 70.0% (+20.0 percentage points). Accounting for only 22% of the full dataset volume, this subset already delivers 57% of the total performance gain.
  • In contrast, the subsequent 3600 hours increase the success rate from 70.0% to 85.0% (+15.0 percentage points), representing an approximately 4.8× reduction in marginal efficiency. We therefore expect that further increases in data volume will yield progressively smaller performance improvements.

Q2:

Q2.1 Regarding your question about manual effort in sim-to-real transfer preparation, most task preparation steps are highly automated, including asset annotation, task configuration, and articulated-object reconstruction. For asset reconstruction, our generate-and-articulate pipeline only needs manual correction for part segmentation refinement. The articulate-and-generate pipeline builds and validates articulated assets automatically from a single reference image and a text prompt. It involves no manual part segmentation or joint configuration. More details can be found in the “others” section of our main response. Overall, the manual effort for preparation is minor.

Q2.2 We do not tune simulation parameters based on real-world evaluation results to mitigate the sim-to-real gap. All asset instances used for real-world evaluation are strictly excluded from the stage-1 data synthesis step. Some of these target instances belong to categories that do not appear in SMART-Data at all. For the remaining instances, we verify their exclusion via manual inspection.

Q3. We answered this question in our response to W2.3. We refer the reviewer to our main response for the definition of our zero-shot sim-to-real transfer setting.


We hope the above responses address your concerns, and we welcome further discussion.

If our responses resolve your concerns, we would greatly appreciate it if you could consider raising your score accordingly.

Comments on the paper

11 comments
AuthorsOpenReview ↗

Main Response

Main Response

We thank the reviewers for recognizing our work and providing valuable feedback.

We identify the following two major concerns that largely shape the current review conclusions:

  1. Concerns regarding the effectiveness and novelty of SMART. To address these concerns, we elaborate on the articulation-aware designs in SMART with richer details. More importantly, we include additional experiments on dataset comparison and ablations of these articulation-related designs to back our conclusions.
  2. Concerns over potential ambiguity in the phrasing “zero-shot sim-to-real transfer”. We clarify the problem formulation of zero-shot sim-to-real transfer in detail and introduce revisions to our manuscript.

We address these two core concerns first.


The novelty and effectiveness of SMART

More details of SMART’s designs

In this section, we aim to elaborate on the designs adopted in SMART.

Our articulation-aware designs are mainly two-fold:

  1. We design part-level annotations and an articulation-aware skill library so that an agent can generate executable articulated-object manipulation tasks.
  • Part-level manipulation annotation. For each articulated object, a VLM identifies the manipulable parts (e.g., handles, knobs), and we annotate their joint types, valid motion ranges, and manipulation semantics. The agent queries these annotations to select actions that are semantically consistent with the target part.
  • Action frames per part. DINO-X localizes each part, from which an anchor action frame is extracted by iterative erosion and 2D-to-3D back-projection. Rule-based sampling around the anchor, conditioned on the joint type, gives a set of feasible action frames.
  • Articulation-aware skill library. Joint types and manipulation semantics are grouped into 23 skills, each with its own constraint types and parameterization. The agent selects skills from the manipulation semantics of the target part and parameterizes them for task execution.
  • Generation procedure. Given a task description, the agent retrieves the robot, objects, and scene with an asset retrieval tool, queries the annotations of the target part, writes a parameterized skill program, and sets the simulation configuration (domain randomization, rendering mode, parallelism).
  1. We specially designed the demonstration generation step to generate articulated-object manipulation trajectories effectively and efficiently.

The articulation-aware designs in SMART primarily focus on contact persistence and constraint adherence, which we elaborate on below:

  • We enforce the generated end-effector trajectories to comply with the articulation constraints of the manipulated parts.
    • Many prior approaches reduce articulated object manipulation to simple contact-based pushing, ignoring the kinematic constraints imposed by joints. For example, opening a microwave door is modeled as pushing it along a straight-line trajectory. In contrast, our method respects these constraints, requiring the entire motion to follow the joint kinematics.
    • We exploit privileged information from simulation to retrieve articulation joint constraints at runtime and generate manipulation motions accordingly.
    • As a component of our basic skill design, we detect substantial displacements of the base links of articulated objects and trigger motion re-generation when such displacements occur. This enables motion adjustments to guarantee constraint following at runtime.
  • We enforce sustained contact throughout in-contact manipulation.
    • We constrain the distance between the action frames defined on the target manipulated part and the end effectors.
    • We equip end effectors with contact sensors to measure scalar contact forces during manipulation and filter out trajectories that produce jerky or extreme contact force feedback.

Both of the above contact-oriented designs are realized via periodic condition checks at runtime.

  • We employ additional techniques to enrich the diversity of our synthetic dataset across both trajectories and visual observations.
    • We introduce perturbations and randomization during motion generation to diversify trajectories.
    • We perform part-level material randomization to vary visual appearances while retaining the semantic structure of visual observations.

In the first version of our manuscript, we focused on how SMART enables agentic task generation. Due to page limits, we provided only a brief description of the articulation-aware design. We will leverage the extra page allowance available in the rebuttal phase to add more details in the revised manuscript.

Our method’s advantage in synthesis efficiency

  1. Our articulation-oriented designs make our agentic task generation feasible and highly automated for articulated objects.
    1. Rather than inferring the articulation structure or contact geometry on its own, the agent selects a skill from annotated manipulation semantics, configures its parameters within the annotated motion range, and samples contact poses from annotated action frames.
    2. To demonstrate the efficiency of our agent-driven method, we report the proportion of automatically generated tasks across different task categories below (the feasibility of the tasks are identified by having success rate or not).
AtomicCompositeLong-horizon
Automatically generated ratio~81%100%~40%
  • Most atomic tasks are generated automatically, and composite tasks are further generated without any human intervention.
  • Long-horizon tasks chain many skills and have a low trajectory success rate, which makes automatic refinement less efficient than manual adjustment in some of the tasks. For tasks that severely suffer from this problem, the agent produces the initial configuration, and a human refines it through the simulator UI.
  1. We show that with our articulation-aware design in the motion generation part, SMART generates articulated-object manipulation tasks efficiently.

The average trajectory success rates in different conditions are reported in the table below.

SMARTSMART w/o constraints
Atomic35%8%
Composite & Long-horizon14%1%
  • Removing the runtime constraint-following designs lowers the trajectory success rate from 35% to 8% on atomic tasks and from 14% to 1% on composite and long-horizon tasks.
  • Without these designs, the motion is planned once and executed open-loop. If the object base shifts during manipulation (e.g., when pulling open a microwave door), the pre-planned motion no longer matches the joint, and the gripper loses contact. Our design detects the displacement and regenerates the motion.
  • The gap is larger on multi-skill (composite and long-horizon) tasks because a task succeeds only if every skill does, so per-skill failures compound.
AuthorsOpenReview ↗

Additional baseline comparison and ablation studies (additional experiments)

We noticed that this section is the most frequently raised point in reviewers’ feedback. To address these concerns, we further provide additional experiments to support the effectiveness of SMART.

Comparative experiments and ablation studies.

We design the experiments as follows:

  • We compare the training performance of SMART-Data with InternData-A1[1] on articulated object manipulation.
    • We take all the articulated object manipulation tasks and some articulated-object-related long-horizon tasks from InternData-A1, which involve 1000 hours of demonstrations, to construct InternData-A1-Art-1000h.
    • To make a fair comparison, we evenly sample 1000 hours of demonstrations from SMART-Data to construct SMART-Data-1000h for comparison.
  • We ablate the designs of articulation-aware motion generation and part-level material randomization. We deactivate these two designs in SMART separately and collect 1000 hours of demonstrations for each ablation variant.
    • For the part-level material randomization ablation, we simply deactivate the fine-grained randomization and perform material randomization without respecting the part material annotations from the asset annotation step.
    • For the articulation-aware design ablation, we deactivate the displacement-driven replanning in our skill design and relax the contact-related runtime checking.

We use these datasets for stage-1 training separately, keeping the training budgets and dataset volume budgets the same.

To evaluate the training effects on articulated-object manipulation tasks, we take the four AC1 tasks in our main experiment to perform stage-2 training and report the success rate of each task.

The results are shown in the table below (we follow the same evaluation protocol as the main experiment):

Data SourceDrawer pulling (%)Microwave Knob Rotating (%)Cabinet Door Sliding (%)Toolbox Lid Flipping (%)Avg (%)
SMART-Data-1000h66.773.366.773.370.0
InternData-A1-Art-1000h73.360.053.346.758.3
SMART-Data-1000h (No-Part-MTL-Rand)53.353.353.366.756.7
SMART-Data-1000h (No-ART-Design)66.760.060.053.360.0

We summarize our observations and analysis as follows:

  • Articulated-object manipulation demonstrations synthesized by SMART yield better training performance than those from InternData-A1 on downstream tasks. This is evidenced by SMART-Data-1000h’s advantage in average task success rate (70.0 vs 58.3).
    • We observe a substantial performance gap on Microwave Knob Rotating, Cabinet Door Sliding, and Toolbox Lid Flipping. These tasks demand persistent in-contact manipulation and strict constraint adherence.
  • The articulated-object manipulation designs in SMART benefit the training performance of the synthesized dataset. Disabling either part-level material randomization or the articulation-aware design degrades model performance (70.0 vs 56.7 vs 60.0).

We further investigate the underlying causes of these observations via failure case analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.