acceptodds
Under review as a conference paper at ICLR 2027

From Parts to Sim: Bottom-Up Autoregressive Generation of Sim-Ready 3D Assets

Abstract

Recent advances in 3D generation have enabled high-quality object synthesis, yet most generated assets remain static, limiting their use in physical simulation and embodied learning. A simulation-ready articulated asset must specify not only its geometry and appearance, but also how its components are organized, how they move, and which physical properties govern their interactions. Existing approaches often recover articulation after geometry generation or represent geometric parts and kinematic links at the same granularity, making it difficult to preserve fine details while maintaining a compact physical structure. We present a bottom-up autoregressive framework that models fine-grained geometric components and rigid kinematic links as distinct but connected levels, enabling the unified generation of geometry, articulation, and simulation-relevant physical attributes, including joint dynamics. To address the scarcity of fully annotated articulated data, we develop a scalable data pipeline and co-train the model using large-scale part-aware geometry and simulation-ready assets. Experiments demonstrate state-of-the-art performance in geometric fidelity, articulation accuracy, and simulation executability. Furthermore, manipulation policies trained on trajectories collected from our generated assets transfer successfully to real-world objects, demonstrating their value for sim-to-real robot learning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Generation, Articulated Objects, Real2Sim2Real

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_33XtOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a bottom-up autoregressive framework designed to generate articulated 3D assets directly usable for simulation from text or image inputs. The framework decouples fine-grained geometric components from rigid kinematic links, thereby retaining detailed geometric features while preventing the kinematic structure from becoming unnecessarily complex. The model is capable of jointly generating component geometries, assembling components into rigid links, and predicting articulation properties as well as simulator-oriented physical attributes including joint damping and stiffness. To address the scarcity of fully annotated articulated data, the authors introduce Mobility-PhyX, a scalable data construction pipeline that integrates geometric and kinematic augmentation techniques with VLM-based physical attribute grounding, and is trained jointly on approximately 430K part-aware objects and 30K simulation-ready assets.

Critical strengths
  1. This paper proposes a novel representation for modeling fine-grained components and jointly generating their corresponding physical information. This representation allows subtle components to be jointly modeled while remaining distinguishable; for example, the caster wheels of a chair can be separated as individual components rather than being merged into a coarse part.

  2. To generate training data and jointly train the model, this paper designs a data construction pipeline for obtaining training data and adopts a heterogeneous training strategy that co-trains on part-aware assets and simulation-ready assets.

Critical weaknesses

None

Questions for authors
  1. I noticed that the authors mention PhysX-Omni, which appears to be a relatively recent method, but it was not evaluated in the experiments. I am not sure what the specific reason is.

  2. In addition, PhysX-3D should be an early work on physically grounded 3D generation, and PhysXNet should be the earliest related dataset serving this line of methods. I think the authors should mention it in the related work.

  3. The experimental section does not include particularly complex visualization examples, so it does not fully demonstrate the advantages of the proposed representation. Moreover, there does not seem to be supplementary material, and I am not entirely sure about the concrete effectiveness of the method.

If the authors can adequately address these concerns, I am willing to increase my rating.

3 comments

We thank the reviewer for recognizing our fine-grained representation and heterogeneous training strategy, and for the constructive suggestions concerning related work and experimental presentation.

  1. Comparison with PhysX-Omni

Our submission includes a discussion and conceptual comparison with PhysX-Omni in Appendix P and Table 7. PhysX-Omni emphasizes unified generation across rigid, deformable, and articulated asset types, whereas our work focuses on separating fine-grained geometric components from rigid kinematic links within articulated objects.

At submission time, PhysX-Omni was an arXiv-only preprint. The ICLR 2027 Reviewer Guidelines ](https://iclr.cc/Conferences/2027/ReviewerGuidelines)[https://iclrc˙c/Conferences/2027/ReviewerGuidelines](https://iclr.cc/Conferences/2027/ReviewerGuidelines)[https://iclr\.cc/Conferences/2027/ReviewerGuidelines explicitly state that “authors are not required to compare to papers solely on arXiv,” while strongly encouraging citation of known related work. We accordingly cited PhysX-Omni and discussed its relationship to our formulation, but did not include it in the numerical comparison.

To address this concern, we have conducted additional experiments comparing our method with PhysX-Omni under the same evaluation protocol detailed in Appendix H. We report the quantitative comparison directly below:

MethodPSNR ↑CD ↓F-score ↑Kinematic score ↑
PhysX-Omni9.0030.050.480.81
Ours12.170.020.520.88
  1. PhysX-3D and PhysXNet

Thank you for pointing out PhysX-3D and PhysXNet. We agree that they are important early contributions to physically grounded 3D generation and will add the corresponding citations to the revised manuscript.

PhysX-3D introduces PhysXGen, a dual-branch framework that jointly models geometry and physical properties within a pretrained 3D latent space. Our method takes a different approach: it explicitly distinguishes fine-grained geometric components from rigid kinematic links and autoregressively generates the component–link–joint hierarchy together with simulator-oriented physical attributes, including joint damping and stiffness. This separation allows multiple geometric components to form one rigid link, preserving geometric detail without unnecessarily increasing kinematic complexity.

Regarding data construction, PhysXNet provides physical annotations through a VLM-assisted human-in-the-loop pipeline, while its PhysXNet-XL extension additionally supports procedural asset composition. Mobility-PhyX focuses on articulated-asset augmentation through retrieval-based component replacement and bidirectional kinematic-tree augmentation. It further grounds joint dynamics by eliciting plausible interaction forces and motion durations from a VLM and deriving damping and stiffness through physical formulations.

  1. Complex examples, supplementary material, and concrete effectiveness

The submitted PDF includes supplementary text and visualizations in its appendices. Complex examples appear in Figures 1 and 4bb, with additional results in Figures 14–15. For example, Figure 1 includes generated assets with 191 geometric components organized into 92 kinematic groups, illustrating the distinction between geometric and kinematic granularity.

Beyond visual examples, Table 5 directly evaluates the generated hierarchy, reporting component F1 of 0.80, grouping ARI of 0.75, joint F1 of 0.84, and tree-edge F1 of 0.90. Figure 5 further demonstrates the utility of generated assets for downstream manipulation-policy learning.

We acknowledge that still images do not fully convey articulated motion or simulation behavior. Full generated asset packages contain high-resolution meshes extracted using marching cube and can readily exceed OpenReview’s 100 MB supplementary-upload limit. We will therefore provide sample videos of complex generated assets in the supplementary materials, allowing their articulation and simulation behavior to be inspected without downloading the full mesh packages.

The author has addressed my main concerns, and I have raised my score accordingly.

Thank you for your follow-up and for confirming that your major concerns have been addressed. We appreciate your constructive feedback and insightful suggestions. Please let us know if you have any further questions or concerns; we would be happy to provide additional clarification.

Reviewer_KEhNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents a bottom-up autoregressive framework for generating simulation-ready articulated 3D assets. It distinguishes fine-grained geometric components from kinematic links, allowing geometric detail and kinematic structure to be modeled at different granularities. Also, this paper proposes to generate joint damping and stiffness parameters for dynamics. Given the scarcity of sim-ready training assets, this paper contributes Mobility-PhyX with augmented assets and physical annotations. Experiments are conducted to demonstrate both geometry quality and simulation executability.

Critical strengths
  1. The disentanglement of geometric and kinematic granularity is well motivated. Grouping multiple independent geometries into one rigid body and generating the sim-ready asset in a functionally bottom-up way retains geometric fidelity and coherent functional relationship.
  2. The VLM-assisted strategy for annotating physical attributes sounds interesting. By asking VLM to provide plausible interaction force or motion duration and deriving corresponding parameters using formulation, it provides a more practical method for annotating the phycical attributes instead of directly predicting the parameters.
Critical weaknesses
  1. The choice of predicting S_{layout} before generating link-wise geometry is insufficiently justified. Given the emphasis on functional grouping, splitting all the entire-object level component layout into each rigid-body group may improve local layout–grouping consistency.
  2. Missing the details about how discrete geometry codes and textual kinematic attributes are tokenized and jointly predicted within the unified autoregressive framework.
Questions for authors
  1. See weakness 1&2
  2. Although co-training improves generated geometric details, it remains unclear whether the part-count distributions differ substantially between the part-aware and simulation-ready datasets and, if so, how this gap affects generalization. For example, how to generate kinect information for a complex asset with plenty of parts.
2 comments

We thank the reviewer for recognizing the motivation behind separating geometric components from kinematic links and the practicality of our physical-attribute grounding strategy. We address the questions below.

  1. Why generate the global layout before link-wise geometry?

Global layout prediction serves as an explicit planning stage: it establishes the component inventory and their spatial arrangement before the model begins generating detailed geometry. Consequently, each link’s geometry can condition on the layout of the entire object, including components that have not yet been generated. Empirically, we observe that removing this planning stage makes the model more susceptible to losing structural coherence during generation, resulting in poorer geometry or missing components. We further include an explicit ablation to quantify this effect:

VariantPSNR ↑CD ↓
With global layout planning12.170.02
Without layout planning8.080.06

Importantly, global planning does not replace functional grouping. After predicting the layout, the model generates geometry in link-wise blocks, with connection-aware spatial DFS within each link and a CreateLink record closing the corresponding component block Section51˙andAppendixC3˙Section 5\.1 and Appendix C\.3. Our design therefore combines object-wide planning with link-local geometric generation.

  1. How are discrete geometry codes and textual kinematic attributes jointly tokenized?

Our geometry tokenization follows MegaParts ](https://arxiv.org/html/2608.14783v1)[https://arxivo˙rg/html/26081˙4783v1](https://arxiv.org/html/2608.14783v1)[https://arxiv\.org/html/2608\.14783v1, whose implementation details are provided in its Appendix A. Each component is represented by an adaptive-length sequence of discrete VQ codes, with codebook entries mapped to dedicated geometry tokens such as <vq48781>. We enlarge the codebook of Qwen to endow the VLM model with the ability to generate those discrete geometry tokens.

These geometry tokens are interleaved with textual asset records, including create_mesh, create_link, and create_joint, within a single autoregressive sequence. Numerical fields—including bounding-box coordinates, mass, joint axes, origins, and limits—are serialized as numerical text and tokenized directly using the Qwen tokenizer. Geometry tokens and textual attributes are thus jointly predicted through next-token prediction, rather than separate generation heads.

Appendix A summarizes the geometry representation, Figure 10 provides a concrete serialized example, and Appendix D describes the joint training objective.

  1. Dataset complexity and generalization to objects with many components

We computed component-count statistics separately for the part-aware and simulation-ready datasets and found broadly similar distributions for samples with fewer than 450 components. Above this range, the part-aware dataset contains an additional high-count tail consisting primarily of multi-object samples, rather than individual articulated objects. We intentionally include these examples as additional geometric supervision to encourage robustness to complex, compositional inputs. We will add the source-separated distributions to Appendix F to make this comparison explicit.

Complex objects use the same model and generation procedure as simple objects: layout planning, link-wise component generation, and joint-record generation. The variable-length representation accommodates additional components while maintaining the distinction between geometric components and rigid links; more geometric components need not introduce correspondingly more joints.

We also clarify that the simulation-ready training corpus includes a privately curated collection of hundreds of complex objects, providing direct training examples of complex geometry and articulation, as briefly mentioned in Section 6.1. Figures 1 and 4bb illustrate complex generated assets. We will supplement more examples with videos to make their articulation and simulation behavior easier to assess.

In our previous response, we explained the motivation for layout-first generation, clarified how geometry codes and kinematic attributes are tokenized and jointly predicted, and discussed component-count distributions and generalization to complex assets. We hope these clarifications address your concerns. Thank you again for your thoughtful feedback and insightful suggestions. Please let us know if you have any remaining questions or would like further clarification.

Reviewer_E5XrOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the problem of generation of simulation-ready 3D assets. The proposed approach recovers geometry and the kinematic tree jointly, along with physical attributes such material properties for rigid links and dynamic properties for joint links. The paper also introduces a data generation pipeline to address the scarcity of sim-ready assets, titled Mobility-PhyX.

The Mobility-PhyX features a dual-level data augmentation process to enrich 3D assets’ part geometry and kinematic structure. First, a part retrieval step takes place, which pinpoints semantically similar parts from external 3D data collections, and replaces selected components from procedural assets, that are generated from the Infinity Mobility method. Next, a kinetic-tree augmentation occurs, that replaces the child or parent kinematic subtree of the component to be replaced, with the corresponding subtree of the newly retrieved part. Finally, physical attributes are assigned to the link and joint level, through a VLM-based annotation pipeline. All these steps, address the issue of part and articulation structure diversity of previous articulated datasets, and provides physical properties annotations, that are crucial for simulation.

The generative model adopts MegaParts’ component-level geometry representation and extends its autoregressive sequence with rigid-link and joint records. Its bottom-up formulation generates fine-grained geometric components in blocks, each closed by a link record that groups its components into one rigid body. Joint records connect these links and specify their motion and behavior. This separation accommodates the different granularities of geometric detail and kinematic structure. The model combines supervision from part-aware and simulation-ready assets, using abundant geometry data alongside the more expensive articulated examples.

Evaluation covers image- and text-conditioned generation on PartNet-Mobility, demonstrating strong geometric fidelity, articulation accuracy, and agreement with expert-validated physical annotations. Additional experiments fine-tune manipulation policies using simulated trajectories collected from generated assets. These policies generalize to unseen simulated instances and transfer to real objects, supporting the utility of the generated environments for robot learning.

Critical strengths
  1. The component-to-link hierarchy addresses an important representational limitation. Separating fine-grained geometry from rigid links allows multiple components to share motion while preserving geometric detail and avoiding unnecessary kinematic complexity. Comparisons with fixed-joint assembly and merged link meshes support this design’s ability to balance geometric fidelity and articulation accuracy. This is crucial because it directly improves the suitability of generated assets for simulation and interaction.

  2. The training strategy makes effective use of scarce simulation-ready supervision. Co-training on part-aware and simulation-ready assets combines broad geometric coverage with supervision for articulation and physical attributes. Mobility-PhyX supports this strategy by expanding geometric and kinematic diversity and supplying plausible physical annotations. Ablations substantiate the training choices where removing co-training substantially degrades geometry quality and surprisingly affects the articulation quality as well.

Critical weaknesses

Damping and stiffness are evaluated against references derived from expert-corrected annotations using the same analytical formulation employed by the data pipeline (Appendix H). These results establish agreement with the proposed annotation framework, but do not independently validate its approximation of real mechanical behavior.

Questions for authors
  1. How are part categories and functional roles matched across datasets, particularly when their labels or category definitions differ?

  2. Beyond support-point alignment, what checks ensure that a replacement remains functionally compatible with the surrounding structure, and how are invalid replacements handled?

4 comments

We thank the reviewer for recognizing the value of our component-to-link hierarchy and the effectiveness of our heterogeneous co-training strategy. We address the concern and questions below.

  1. Gap between real mechanical behavior and the data pipeline of Stiffness and Damping
GeneratedReal-world
Classjoint-typeDampingStiffnessDampingStiffness
microwavedoor-revolute0.4500.430
fridgedoor-revolute1.9902.050
laptopscreen-revolute4.2904.7380

Our purpose in introducing physical parameters is to make generated assets directly usable in physics simulation. As clarified in the paper, we aim to generate parameters within physically plausible operating ranges, rather than claim exact recovery of instance-specific mechanical properties. Our damping and stiffness formulations follow standard mechanical relationships between force and velocity, torque and angular velocity, and restoring force and displacement. Under the stated modeling assumptions, these formulations provide physically interpretable and consistent effective parameters, yielding physically grounded supervision for training. The downstream sim-to-real experiments further support the practical utility of the resulting assets for learning transferable manipulation policies.

To further validate these parameters against real-world measurements, we measured the joint damping and stiffness of three physical objects: a microwave, a refrigerator, and a laptop. Then we used photographs of these same objects as inputs to our generative model to produce their corresponding simulation assets.

As shown in the table, the generated damping coefficients closely match the measured values. The generated stiffness values also agree with the measured zero stiffness in all three cases. These results provide additional empirical evidence that our pipeline produces joint parameters consistent with the measured mechanical properties of the tested real-world objects.

  1. How are part categories and functional roles matched across datasets, particularly when their labels or category definitions differ?

As described in Appendix E.1.1, part geometry augmentation retrieves parts from external 3D asset collections with the same semantic category. For kinematic-tree augmentation, Appendix E.1.2 specifies that corresponding joints are selected by matching their child links: the child links must share similar semantic and functional roles, and their corresponding joints must have compatible motion types.

To clarify how these criteria are implemented in practice, our augmentation pipeline uses the original procedural assets as the reference for replacement. The external articulated-object datasets used as sources for our data augmentation largely consist of manually authored assets. Within a given source dataset and object category, these assets tend to share similar kinematic-tree structures, part-level semantics, and functional roles. This regularity provides a practical basis for organizing human review by source dataset and object category.

Specifically, before applying the matching criteria, we manually check whether the parts or kinematic structures within each source dataset/category are suitable for replacing the corresponding parts or structures in the original procedural assets. Their suitability is assessed relative to the semantic and functional roles in the target procedural templates, allowing correspondences to be established despite differences in source labels or category definitions. Only parts and kinematic structures deemed suitable through this review are considered for augmentation, and the matching criteria described above are subsequently applied within this restricted candidate set.

  1. Beyond support-point alignment, what checks ensure that a replacement remains functionally compatible with the surrounding structure, and how are invalid replacements handled?

Beyond support-point alignment, we maintain replacement validity through three complementary stages.

First, our asset construction follows a procedural pipeline in which category-specific object layouts, dimensions, and articulation parameters are explicitly controlled by human-defined rules. These rules constrain the structural organization of the assets from the generation stage. This follows the procedural generation paradigm used in systems such as Infinigen[1] and Infinite Mobility[2].

Second, our part and kinematic-tree replacements serve as data augmentation within this procedural framework. As clarified in our response on cross-dataset matching, only suitable candidates are considered for replacement. Beyond support-point alignment, the retrieved geometries are scaled and reassembled according to the predefined object layout and the procedural generator’s geometry-aware placement rules. As described in Appendix E.1.2, joint origins, motion axes, and motion ranges are reconstructed after reassembly, with motion limits further constrained by collision conditions on the resulting geometry. We additionally conduct manual spot checks of the augmented assets to assess replacement validity.

Finally, we apply VLM-based visual-quality filtering to identify and discard low-quality corner cases. Assets rejected by this filtering stage are excluded from the training corpus. This provides an additional quality-control layer alongside procedural constraints, candidate screening, and manual inspection.

[1]Raistrick A, Lipson L, Ma Z, et al. Infinite photorealistic worlds using procedural generation[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023: 12630-12641.

[2]Lian X, Yu Z, Liang R, et al. Infinite mobility: Scalable high-fidelity synthesis of articulated objects via procedural generation[J]. arXiv preprint arXiv:2503.13424, 2025.

Thank you again for your thoughtful feedback. As the discussion period draws to a close, we would like to briefly follow up on our previous response, in which we presented experimental comparisons of generated and real joint attributes, clarified the dataset-construction process, and described safeguards for ensuring functional compatibility. We hope these results and clarifications address your concerns, and we would welcome any remaining questions or requests for clarification. Thank you for your time and consideration.

I want to thank the authors for the detailed response. They have addressed my concerns, esp. the gap between real mechanical behaviour and the data pipeline of stiffness and damping. I will retain my initial positive assessment.

Comments on the paper

1 comment

Dear Reviewers,

We sincerely thank you for your careful reviews and insightful suggestions. We have updated the manuscript in response to your feedback and marked the revised text in blue to make the changes easy to identify.

We welcome any further questions or concerns and would be happy to discuss them and provide additional clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.