acceptodds
Under review as a conference paper at ICLR 2027

FITR: From Articulated Object Images to Interactive 3D Assets for Real-World Manipulation

Abstract

Collecting real-world training data for articulated object manipulation is costly, motivating scalable data generation in simulation. Recent advances in 3D asset generation have improved visual fidelity and geometric quality, but supporting robot interaction also requires appropriate physical dimensions, contact points and directions, and joint dynamics. To this end, we introduce FITR, a pipeline that constructs interactive 3D assets from images of articulated objects for policy learning in simulation and zero-shot deployment on real robots. We construct FITR-Assets by augmenting articulated assets with absolute scale, affordances, and calibrated joint drive parameters, and evaluate annotation quality on FITR-Bench. Using these assets, we combine motion planning with domain randomization to automatically collect approximately 60k manipulation demonstrations in Isaac Sim. Then, we fine-tune vision-language-action (VLA) policies via imitating the demonstrations and RL-driven simulated post-training, enabling zero-shot transfer to real world. Through systematic ablation studies and comparative experiments in simulation and on real robots, we demonstrate that FITR enables zero-shot sim-to-real transfer for articulated object manipulation. We release FITR-Assets, the simulation demonstration dataset, policy checkpoints, and source code.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Articulated Object Manipulation, Sim-to-Real Transfer, Simulation-Ready Assets

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Tp8DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FITR is a pipeline that takes articulated-object images, generates assets with an existing image-to-URDF method (URDF-Anything+), and then augments them with metric scale, contact points, and joint drive parameters. Scale comes from a VLM prompted with same-category reference dimensions. Contacts are derived geometrically from joint axes, and drive parameters are category-level baselines calibrated against force-gauge measurements. The resulting 2,772 assets (FITR-Assets) and a 240-object benchmark (FITR-Bench) support about 60k motion-planned demonstrations in Isaac Sim. Three VLAs (π0.5, X-VLA, PRTS) are fine-tuned on these demonstrations, and PRTS is further improved with residual RL and distillation (PLD). The authors report zero-shot real-robot transfer on six tasks, with mean NJC around 57% for π0.5 and 82.5% for post-trained PRTS.

Critical strengths
  1. The paper is an end-to-end pipeline from image to asset to demonstrations to RL post-training to real-robot deployment, and the real-world results show nonzero zero-shot progress on all six tasks. This makes the work practically useful and goes beyond asset-quality metrics.

  2. The attribute completion is cheap and effective, with reference-guided scaling cutting sMAPE from 27.75% to 7.93% and geometric contacts cutting error from 21.45 cm to 1.48 cm. The released assets, demonstrations, and code add value for the community.

Critical weaknesses
  1. The central claim that physical augmentation (especially joint drive calibration) enables sim-to-real transfer is never directly tested as far as I can tell. There is no ablation with and without calibrated drive parameters or absolute scale in terms of policy performance, and the only policy ablation (Table 2) confounds randomization, dataset size (10k vs 300 demos), and training steps (30k vs 5k).

  2. The evaluation is limited and the novelty is modest. Real-world results cover only six single-object tasks with 20 trials each and no variance or confidence intervals, and the PRTS RL gains are measured against a baseline from the same sim distribution. The contact derivation is a simple analytic rule, and the drive parameters are category-level constants from small measurement sets, so the contribution is mostly integration of existing components (URDF-Anything+, ArticuBot-style planning, PLD).

Questions for authors
  1. Can you ablate policy performance with uncalibrated or default drive parameters and with wrong scale, holding the dataset size and training budget fixed? This is the key evidence that the physical attributes matter for transfer.
  2. Can you provide variance or confidence intervals over seeds and trials, and compare against baseline assets such as PartNet-Mobility with default physics? Also, how often does the VLM-completed instance-level drive adjustment actually differ from the category baseline?
  3. (Clarification) Table 1 is not clear. Why is handle accuracy only computed for 5 categories? Also, how are contact points validated beyond successful planning in simulation?
1 comment
AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer Tp8D,

As the discussion period draws to a close, we would welcome your feedback on our responses. We have added six-task physical attribute ablations, training-step comparisons, and standard deviations and confidence intervals for selected simulation results. We also clarified the training settings and evaluation protocol and provided a two-stage manipulation demo.

Please let us know whether any concerns remain after considering this additional evidence. We would be happy to discuss them further. Thank you again for your time and consideration.

Reviewer_3JVWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FITR, a method that augments articulated assets with metric scale, actionable contact points and equivalent joint-drive parameters. FIFR uses same-category references for scale estimates and contacts are derived from geometry and joint kinematics and drive parameters combine physical measurements on representative objects with completion by VLM. The experiment results show that the pipeline supports VLA fine-tuning followed by residual-RL recovery-data collection and distillation

Critical strengths
  • The paper addresses the gap between visually plausible assets and assets useful for contact-based policy learning.
  • Tab 1 reports substantial improvements over direct VLM estimation. The use of known geometry and kinematics is a reasonable inductive bias.
  • The pipeline is evaluated through downstream real-robot interaction, not limited to asset appearance. The appendices provide useful details on force measurement, joint-drive units, randomization, and the policy execution interface.
Critical weaknesses
  • The distinctive physical-dynamics contribution lacks direct validation and controlled downstream attribution. There is no quantitative evidence that VLM-derived instance adjustments improve on the calibrated category baseline. The manipulation studies compare backbones, overall training recipes, and residual-RL post-training; they do not establish which gains require the proposed physical completion.
  • The independence and provenance of the asset evaluation are not clear. Appendix B defines scale ground truth using the corresponding physical object while Appendix A states that most assets originate from URDF-Anything+ and the remainder from PartNet-Mobility. It does not sufficiently describe how each benchmark mesh is associated with a measured physical counterpart and how reference objects are separated from evaluation objects, or how contact-point ground truth is defined when several contacts are valid.
Questions for authors

Please see above. Additionally,

  • On existing tasks, what changes when using default drives, measured category-level drives, and VLM-adjusted instance drives, while holding the rest of the pipeline fixed?
  • How were physical ground-truth counterparts and contact labels obtained for FITR-Bench? Please specify prediction-versus-label independence, reference/evaluation overlap, and the number of real test instances that were not used for calibration or asset construction.
3 comments
AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer 3JVW,

As the discussion period draws to a close, we would like to follow up on our response. We have added a six-task comparison of VLM-predicted and measured category-level joint-drive parameters, and clarified benchmark provenance, physical-object matching, annotation independence, and calibration overlap.

We would appreciate hearing whether these results and explanations address your concerns, or whether further clarification would be helpful. Thank you again for your thoughtful feedback and time.

Thank the author for the detailed rebuttal. These responses addressed my major concerns and I maintain the recommendation as Weak acceptance.

Thank you for your thoughtful review and follow-up. We are glad that our responses addressed your major concerns, and we appreciate your time and constructive feedback.

Reviewer_X5MEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents FITR, a pipeline that converts articulated object images into interactive 3D simulation assets. By augmenting the assets with physical scale, contact points, and joint dynamics, the method automatically generates demonstrations in simulation and trains VLA policies through imitation learning and reinforcement learning, enabling zero-shot sim-to-real transfer.

Critical strengths

S1. A practically valuable and well-motivated pipeline.

The paper addresses an important gap between image-generated articulated assets and their practical use in robotic manipulation. By integrating physical augmentation, simulation demonstration generation, and policy training, FITR provides a complete pipeline for zero-shot sim-to-real transfer.

S2. Substantial experimental effort and encouraging real-world results.

The paper constructs 2,772 articulated assets and approximately 60K successful simulation demonstrations. Evaluation across three VLA backbones and six manipulation tasks provides encouraging evidence of zero-shot real-world transfer. The improvements from simulation-based post-training are particularly notable.

Critical weaknesses

W1. Limited methodological novelty.

The pipeline largely builds upon existing techniques. Articulated asset generation relies on URDF-Anything+ (Wu et al., 2026); physical attribute modeling has been explored in PhysX-Omni (Cao et al., 2026); articulation-aware contact selection and demonstration generation are closely related to FlowBot3D (Eisner et al., 2022) and ArticuBot (Wang et al., 2025). Large-scale simulation data generation and domain randomization have also been investigated in RoboGen (Wang et al., 2024) and RoboTwin 2.0 (Chen et al., 2026), while residual RL with SFT distillation follows PLD (Xiao et al., 2026). Although the integration has engineering value, a distinctive methodological contribution remains difficult to identify.

W2. Limited generality of contact point derivation.

The heuristic contact point derivation may not generalize across diverse articulated objects. For example, maximizing the moment arm does not necessarily yield suitable contacts for scissors, folding knives, or articulated lamps, potentially limiting the scalability of the pipeline. Moreover, deterministic contact points may reduce demonstration diversity and policy generalization.

W3. Unclear ground-truth annotation protocol.

The ground-truth annotation protocol for contact points in Table 1 is insufficiently described. Although manual verification is mentioned, the criteria for valid contacts and the independence of the annotations from the proposed method remain unclear. Additionally, multiple contact points may be equally valid, raising concerns about pointwise MAE evaluation.

W4. Insufficient validation of the core physical augmentation contributions.

The paper lacks controlled end-to-end ablations isolating the contributions of absolute scale recovery, contact point derivation, and joint drive calibration. Most downstream experiments, including VLA comparisons, domain randomization, and RL post-training, primarily demonstrate the effectiveness of the established techniques rather than the proposed physical augmentation components.

[1] Wu et al., 2026. URDF-Anything+: Autoregressive Articulated 3D Models Generation for Physical Simulation. arXiv:2603.14010. https://arxiv.org/abs/2603.14010

[2] Cao et al., 2026. PhysX-Omni: Unified Simulation-Ready Physical 3D Generation for Rigid, Deformable, and Articulated Objects. arXiv:2605.21572. https://arxiv.org/abs/2605.21572

[3] Eisner et al., 2022. FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects. RSS 2022. https://roboticsconference.org/2022/program/papers/018/

[4] Wang et al., 2025. ArticuBot: Learning Universal Articulated Object Manipulation Policy via Large Scale Simulation. RSS 2025. https://arxiv.org/abs/2503.03045

[5] Wang et al., 2024. RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation. ICML 2024. https://proceedings.mlr.press/v235/wang24cc.html

[6] Chen et al., 2026. RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation. ICML 2026. https://proceedings.mlr.press/v306/chen26j.html

[7] Xiao et al., 2026. Self-Improving Vision-Language-Action Models with Data Generation via Residual RL. ICLR 2026. https://arxiv.org/abs/2511.00091

**W5. Limited real-world evaluation **Despite constructing 2,772 assets across 12 categories, real-world evaluation is limited to six single-arm articulated manipulation tasks, primarily involving simple pushing and rotating motions. More complex contact-rich and multi-stage manipulation tasks are not evaluated.

**W6. ** Although the paper emphasizes automatic construction of interactive assets from images, the pipeline still relies on physical measurements of object dimensions and operating forces, as well as manual verification of contact points. These requirements may limit its scalability and degree of automation.

Questions for authors

see weakness

1 comment
AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer X5ME,

As the discussion period draws to a close, we would appreciate your feedback on whether our responses address your concerns. We have added six-task ablations of contact-point derivation, absolute scale, and joint-drive calibration, clarified FITR’s contributions and annotation protocol, and provided a two-stage manipulation demo.

Please let us know if any questions remain. We would be happy to clarify further. Thank you again for your time and consideration.

Comments on the paper

1 comment

Dear reviewers, the discussion period is ending. Could you please engage with the authors’ rebuttal? Thanks, AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.