acceptodds
Under review as a conference paper at ICLR 2027

ASSET2SIM: Automatically Improving Simulation Physics of Articulated Objects

Abstract

Approximate physics simulators have emerged as a key resource for training and evaluating robotic policies in simulation, especially for locomotion and navigation capabilities. A key bottleneck to realizing their utility for manipulation tasks is the dearth of assets (objects and environments) compared to the real world. Today's largest repositories of object assets focus on rigid objects offering limited affordances for manipulation beyond global repositioning. On the other hand, articulated objects which afford more nuanced manipulation (e.g., staplers, trashcans, cabinets, microwaves, folding knives) remain largely absent from simulators. Widely available curated libraries of articulated assets are limited: (1) they are either geometry-only, lacking annotations for physics properties like link inertias and joint frictions, or (2) they cover a limited number of assets designed with extensive manual effort and still have noisy physics annotations, as we show. We present Asset2Sim, an autonomous vision-language model-guided procedure to improve the physics properties of articulated object assets for sim-readiness. Asset2Sim utilizes multiple render views, geometric data, semantic annotations, and volumetric properties to iteratively refine estimates of an articulated asset's physics properties and initial states. We also develop an evaluation protocol to thoroughly validate the physics fidelity, RL training-readiness, and initialization stability of Asset2Sim assets in robot manipulation tasks, establishing over higher mean sim-to-real correlation and over higher average policy training success rate over the next-best baselines. We will release the Asset2Sim-X library of 1892 sim-ready articulated object assets spanning 44 categories, as well as our code to enable scaling such repositories further in the future.

Primary area:
applications to robotics, autonomy, planning
Keywords:
articulated objects, 3D asset repair, robot manipulation simulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QpWdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Asset2Sim, a pipeline that uses vision-language models (VLMs) to improve the simulation readiness of articulated URDF assets. Given an asset, the method uses multi-view renderings and mesh geometry to estimate physical properties such as mass, inertia, friction, damping, and scale. It then checks the predicted parameters for physical validity and uses simulator feedback to refine the initial joint configuration when self-penetration is detected, while attempting to preserve the intended object state. The paper evaluates the approach in terms of initialization stability, perceived physical realism, robot policy performance, and sim-to-real correlation. Comparisons with VLM-based and human-annotated baselines are conducted on articulated manipulation tasks, including evaluations with VLA policies and state-based reinforcement learning. The authors also plan to release a collection of refined articulated assets based on PartNet-Mobility.

Critical strengths

1.I think the paper addresses an important problem for robot manipulation. Existing articulated object assets often have usable geometry and joint structures, but their physical parameters are not necessarily suitable for simulation. Improving these assets directly could save considerable manual effort when setting up manipulation environments. I also find the focus on refining existing URDFs more practical than building new assets from scratch.

2.One strength of the paper is that it evaluates the assets beyond whether they are physically valid at initialization. The experiments also examine interaction realism and downstream policy performance, including VLA-based manipulation and state-based RL training. These evaluations provide a more useful picture of simulation readiness than static checks alone. The planned release of Asset2Sim-X, with 1,892 assets across 44 categories, could also be valuable to the robotics community if the assets prove reliable in downstream use.

Critical weaknesses

1.I am not entirely sure how simulator feedback is used to refine the physical parameters. The paper describes how the VLM estimates properties such as mass, inertia, friction, and damping, but Section 2.4 seems to focus on optimizing the initial joint configurations to reduce self-penetration. It is therefore unclear whether the physical parameters themselves are updated iteratively based on simulation results, or whether they are estimated first and then kept fixed.

2.I am not fully convinced that the reported improvements come from more accurate physical parameters. The method also adjusts the initial joint configurations, but I could not find an ablation separating the effects of these two components. A better initial configuration alone might improve stability and task success without necessarily making the dynamics more realistic.

Questions for authors

1.The source of the reported improvements in simulation stability and manipulation performance is unclear. The current experiments do not seem to isolate the effects of physical property estimation from those of initial joint configuration refinement. As a result, it is difficult to determine whether the gains are primarily attributable to more accurate physical parameters, better initial configurations, or both.

2.The reported greater-than-2.12x improvement in mean sim-to-real correlation is confusing to me. Using a simple arithmetic mean of the three VLA SRCC values in Table 2 gives approximately 0.513 for Asset2Sim and 0.267 for the strongest baseline, Human (GenSim2), corresponding to about 1.93x rather than 2.12x. It is unclear how the reported 2.12x figure is obtained.

2 comments
AuthorsOpenReview ↗

Rebuttal (1/1)

Thank you for the feedback, and for recognizing the utility of our approach for improving robotics simulation design.

Simulator-in-the-Loop Refinement

We update the positions of the focus joints (defined in lines 194-196), which are configurations in the feasible joint space (defined in lines 188-190). Thus, we only updating the kinematic properties of the joints during refinement for the subset of joints in Jfocus(t)J_{\text{focus}}^{(t)}. The other properties (mass, inertia, friction, damping, etc.) are generated in a single pass, as shown in Figure 1.

We respectfully disagree that we do not isolate the effects of physical property estimation from joint configuration refinement: our Direct VLM baseline is equivalent to running the Asset2Sim pipeline without any iterative simulator-guided refinement. The Direct VLM baseline uses the same physical properties as Asset2Sim alongside the VLM’s one-shot estimated joint positions. The performance difference between Asset2Sim and Direct VLM can hence be reasonably attributed to the usage of iterative refinement for optimizing joint configurations. Additionally, our experiments on the GenSim2 assets use human-assigned object size and joint configurations alongside default simulator values for the remaining properties. The difference between GenSim2 and Asset2Sim results quantifies the impact of correct physical property estimation, as joint configurations are optimized for low interpenetration in both cases.

SRCC Score Calculation

Our original estimate reported the improvement in mean SRCC across all environments between GenSim2 and Asset2Sim for π0.5\pi_{0.5}, which is the median result in terms of performance increase between the 3 evaluated VLAs. We have incorporated your feedback to update this estimate to reflect the mean of improvement in mean SRCCs across all 3 VLAs, and updated our paper to use an updated 1.92x figure.

Dear Reviewer QpWd,

Thank you for your time and effort in providing your thoughtful review. As the discussion period is ending soon, we would like to gently follow up on our rebuttal response. We are happy to provide any further clarifications if any of our points remain unclear, or if you have any further questions.

Thank you!

Reviewer_9aeMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes: "Asset2Sim", an approach for automatically improving the simulation physics of a base articulated object asset; and an evaluation protocal for "sim-readiness".

The method: A VLM proposes an initial "complete" asset representation from a base URDF, multi-view rendered images, and additional mesh properties. This initial proposal is then iteratively refined: a VLM is repeatedly prompted to update the joint specifications, and the updates are accepted when they decrease penetration.

The evaluation protocol: The "sim-readiness" protocol defines a family of scoring criteria, covering: initialization stability (penetration, pose stability, and joint oscillation); VLA+VLM and human judgements; success rates of policies trained on those objects and evaluated on the same objects; deviation from expert annotations; and alignment with user prompts.

Critical strengths

The iterative refinement pipeline proposed is a practical method for improving articulated assets for robotic simulation. The proposed evaluation protocol is also straightforward and practical. I believe this paper is of reference value for researchers and practitioners building simulated environments with articulated objects.

Critical weaknesses

The methodological contribution (the iterative improvement of articulated URDF) is limited in both scope and novelty. If I correctly understand Sec. 2, during the refinement iterations, only the joint's kinematic parameters can be updated, and only the penetration score is used as the quantitative feedback. What this refinement can do is rather limited compared to the goal of "sim-readiness"; it does not address many other practical issues, especially dynamic properties like friction, stiffness, and damping (as already pointed out by the proposed evaluation protocol). On the novelty side, prior work has been using VLMs for asset refinement and even whole-scene generation [1].

The proposed evaluation protocol is also limited in novelty. I understand it as a list of metrics that have been used by the community to measure simulation fidelity, e.g. SRCC as in SIMPLER and PolaRiS, initialization stability as in Molmospaces, human ratings (many others).

Some experiments are not very statistically meaningful. For example, to evaluate SRCC, each real-world success rate seems to be computed over only 5 rollout episodes (Line 272).

Other smaller issues: the paper's presentation could be improved. For example, in Figure 1, much of the text is too small to read without ~3x zoom. Sec 2.1 is a bit overloaded with notation, and many symbols defined are used only once here. I also find Sec 3 a bit tedious to read, with many details regarding each score definition.

[1] Pfaff et al., "SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes". ICML 2026.

Questions for authors
  1. "Simulation in the loop": is the usage of simulation here only for detecting contacts and computing penetration depth, without stepping the dynamics?

  2. During the iterative improvement, what parameters exactly are updated, and what target metrics are used (and how)? Is my understanding in "Critical Weaknesses" above correct? (If not, please correct me!)

  3. "Multi-view visual rendering": what joint state is used to render them during iterative refinement?

3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Thank you for the feedback, and for recognizing the utility of our approach for improving robotics simulation design.

Novelty of Iterative Refinement

We do not claim that our core contribution involves refining LLM/VLM outputs using geometric or rendered feedback for scene creation, and in fact, we cite other environment creation methods (e.g. Robogen [1] (line 36)) which validate their environment design pipelines using simulator or rendered feedback.

A critical difference between these environment-generation methods and our approach, is that they retrieve assets from prior large-scale asset libraries (e.g. Robogen from Objaverse [3] and PartNet-Mobility [4], SceneSmith [2] retrieves from ArtVIP [5]). The primary focus of our paper is making articulated assets in the asset libraries interaction-ready without extensive manual effort or relying on simulator/library defaults, as several such scene-generation methods do.

Previous efforts in this direction involve image/text-to-URDF methods like PhysX-Anything [6] and URDF-Anything [7], which claim that their pipeline-generated URDFs are “simulation-ready” despite not utilizing simulator feedback like our method. Our argument is that training or evaluating policies on such pipeline-generated assets, which do not assign relevant physics properties for realistic simulations, will diverge from real-world performance and undermine simulation training success rates. Our contribution is demonstrating that such simulator-based refinement is critical for determining the interaction readiness at the asset level, not the scene level, and showing that refinement improves sim-to-real-correlation alongside other asset evaluation metrics. Our novelty lies in how we instantiate the feedback loop and leverage the penetration score, and we believe our results are of interest to the community because they demonstrate that such simulator-based asset refinement can lead to improvements in sim-to-real correlation of policies and human realism evaluations of robot-asset interactions.

With respect to SceneSmith, they retrieve articulated assets from ArtVIP, a smaller articulated-asset dataset which required extensive human labor to carefully craft digital twins of real-world articulated objects. SceneSmith also primarily uses the VLM for retrieving relevant assets, and for text-to-URDF construction, with one-shot VLM assignment only of limited physical properties. SceneSmith describes articulated assets from PartNet-Mobility as having “visual and joint quality insufficient for realistic scene generation” [2], and thus requires a higher-quality asset as a starting point for their method. This underscores the problem we are attempting to solve: improving the realism of lower-quality articulated asset URDFs with missing properties at scale to improve their suitability for evaluation and training of robot manipulation policies. We have now updated our paper with citations to SceneSmith to clarify this difference (lines 607-611).

Evaluation Protocol Novelty

While we acknowledge that SRCC and initialization stability have been used as metrics to evaluate overall robotics simulation environment quality, these metrics have not yet been widely adopted in papers involving “simulation-ready” URDF asset design (e.g. PhysX-Anything, URDF-Anything). We claim novelty with respect to an articulated asset evaluation protocol for interaction-readiness (as stated in lines 101-102) - to the best of our knowledge, we are the first URDF refinement paper to evaluate assets using a protocol that comprises all of the following: VLA-based SRCC, reinforcement learning-based policy training, human evaluations, and simulation stability evaluations.

Simulator-in-the-loop Refinement: Yes, we do not step through the dynamics of the simulator during iterative refinement and only leverage penetration scores. We update the positions of the focus joints (defined in lines 194-196), which are configurations in the feasible joint space (defined in lines 188-190). Thus, we only update the kinematic properties of the joints during refinement for a subset of joints.

Multi-View Visual Rendering: We render the updated asset (with the updated joint state) at each iteration of simulation-in-the-loop refinement (line 205).

Statistical Significance: We acknowledge that while the number of rollouts may be small, these are due to limitations in hardware usage as well as due to our efforts to manually evaluate each rollout for task success. Please note that we evaluate 5 rollouts for each of our 4 methods (plus the real-world VLA) for 11 environments, which results in manual evaluation of 275 rollouts. We believe that evaluating SRCC at a larger scale is difficult in an academic setting with a small team, without a team size comparable to an industry team.

AuthorsOpenReview ↗

Rebuttal (2/2)

References

  1. Wang et al. (2023). Robogen: Towards unleashing infinite data for automated robot learning via generative simulation.
  2. Pfaff et al. (2026). Scenesmith: Agentic generation of simulation-ready indoor scenes.
  3. Deitke et al. (2023). Objaverse-XL: A Universe of 10M+ 3D Objects.
  4. Xiang et al. (2020). SAPIEN: A simulated part-based interactive environment.
  5. Jin et al. (2025). Artvip: Articulated digital assets of visual realism, modular interaction, and physical fidelity for robot learning.
  6. Cao et al. (2025). Physx-anything: Simulation-ready physical 3d assets from single image.
  7. Li et al. (2025). Urdf-anything: Constructing articulated objects with 3d multimodal language model.
  8. Le et al. (2024). Articulate-anything: Automatic modeling of articulated objects via a vision-language foundation model.

Dear Reviewer 9aeM,

Thank you for your time and effort in providing your thoughtful review. As the discussion period is ending soon, we would like to gently follow up on our rebuttal response. We are happy to provide any further clarifications if any of our points remain unclear, or if you have any further questions.

Thank you!

Reviewer_oFN1OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors design Asset2Sim, a system to improve articulation parameters with feedback from VLMs. The system starts by providing the VLM with mesh statistics, annotations and renders, then iteratively requests revisions for any collisions between parts. The authors use this to construct Asset2Sim-X, which uses their pipeline to create new physics properties for a set of PartNet-Mobility assets. This dataset improves policy performance in sim tasks.

Critical strengths

The work curates a new set of physics parameters for PartNet-Mobility which is broadly useful and seems to improve policies in simulation.

Critical weaknesses

The method is not surprising or insightful to me, the loop of a VLM recieving feedback from geometric checks is common for other tasks e.g indoor arrangement in SceneWeaver (Neurips25) and similar. I am unsure of gemini2.5flash's ability to make realistic decisions about parameters or to consult image inputs, or that Haiku4.5 can be an effective baseline.

The authors should compare to the PhysX-Mobility dataset, which also has improved parameters for PartNet-Mobility, albeit lacks some provided by the authors.

Questions for authors

How meaningful are the simulation parameters besides satisfying the inter-penetration stopping criterion?

How many and which assets are marked as allowed to self-collide? Are the baselines also marked to self-collide for some or many assets? I believe this might be a relatively common way to use the dataset, since the geometry is not high quality but self-collision often doesnt pose an issue.

Can Asset2Sim-X be compared to PhysX-Mobility?

3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Thank you for the feedback, and for recognizing the utility of our approach for policy training.

Novelty of Iterative Refinement

We do not claim that our core contribution involves refining LLM/VLM outputs using geometric or rendered feedback for scene creation, and in fact, we cite other environment creation methods (e.g. Robogen [1] (line 36)) which validate their environment design pipelines using simulator or rendered feedback. We have now updated our paper with citations to SceneWeaver [2] and clarified this difference (lines 604-607).

A critical difference between these environment-generation methods and our approach, is that they retrieve assets from prior large-scale asset libraries (e.g. SceneWeaver retrieves from Objaverse [3] and Infinigen [4], Robogen from Objaverse [3] and PartNet-Mobility [5]). The primary focus of our paper is making articulated assets in the asset libraries interaction-ready without extensive manual effort or relying on simulator/library defaults, as several such scene-generation methods do.

Previous efforts in this direction involve image/text-to-URDF methods like PhysX-Anything [7] and URDF-Anything [8], which claim that their pipeline-generated URDFs are “simulation-ready” despite not utilizing simulator feedback like our method. Our argument is that training or evaluating policies on such pipeline-generated assets, which do not assign relevant physics properties for realistic simulations, will diverge from real-world performance and undermine simulation training success rates. Our contribution is demonstrating that such simulator-based refinement is critical for determining the interaction readiness at the asset level, not the scene level, and showing that refinement improves sim-to-real-correlation alongside other asset evaluation metrics. Our novelty lies in how we instantiate the feedback loop and leverage the penetration score, and we believe our results are of interest to the community because they demonstrate that such simulator-based asset refinement can lead to improvements in sim-to-real correlation of policies and human realism evaluations of robot-asset interactions.

Utility of Non-Interpenetration-Associated Parameters

We find that it is often not sufficient to simply minimize inter-penetration to succeed in our valuation protocol. We have provided a visual reference in our supplementary material: despite low interpenetration, both the GenSim2 assets fail during policy evaluation due to exceedingly high joint stiffness while Asset2Sim assets succeed. Similarly, inaccurate or missing physical properties PP (lines 151-154) can adversely affect robot-asset interactions, despite not being updated in the refinement loop.

VLM Estimate Choice

We use Gemini-2.5-Flash as our VLM since we edit assets at a much larger scale, and it would be prohibitively expensive to use more advanced models. Articulated asset design has been shown to work with less sophisticated models in previous work (e.g. Articulate-Anything [9] uses Gemini-1.5-Flash while PhysX-Anything [7] uses Qwen2.5-VL), so we believe that our results are consistent with prior work and justify our VLM choice.

Comparison with PhysX-Mobility

We do not believe that PhysX-Mobility is an appropriate comparison for Asset2Sim, since the PhysX-Anything method focuses on URDF generation and part segmentation, and has limited overlap with properties that Asset2Sim assigns to a pre-existing URDF. Their metrics primarily focus on image-to-URDF generation, measuring faithfulness to a reference image, while we measure simulation-readiness via the successful training of policies and high sim-to-real correlation with real-world VLA evaluation. If we imported PhysX-Mobility assets into our scenes, we would have to rely on simulator defaults to assign several parameters for evaluating policies, likely leading to worse performance than the human-assigned parameters in our GenSim2 baseline.

Self-Collision

We permit self-collision for the articulated assets in all our environments for all baselines and our methods during our RL and VLA policy evaluations for a fair evaluation, as is the standard approach.

AuthorsOpenReview ↗

Rebuttal (2/2)

References

  1. Wang et al. (2023). Robogen: Towards unleashing infinite data for automated robot learning via generative simulation.
  2. Yang et al. (2025). Sceneweaver: All-in-one 3d scene synthesis with an extensible and self-reflective agent.
  3. Deitke et al. (2023). Objaverse-XL: A Universe of 10M+ 3D Objects.
  4. Raistrick et al. (2024). Infinigen indoors: Photorealistic indoor scenes using procedural generation.
  5. Xiang et al. (2020). SAPIEN: A simulated part-based interactive environment.
  6. Jin et al. (2025). Artvip: Articulated digital assets of visual realism, modular interaction, and physical fidelity for robot learning.
  7. Cao et al. (2025). Physx-anything: Simulation-ready physical 3d assets from single image.
  8. Li et al. (2025). Urdf-anything: Constructing articulated objects with 3d multimodal language model.
  9. Le et al. (2024). Articulate-anything: Automatic modeling of articulated objects via a vision-language foundation model.

Dear Reviewer oFN1,

Thank you for your time and effort in providing your thoughtful review. As the discussion period is ending soon, we would like to gently follow up on our rebuttal response. We are happy to provide any further clarifications if any of our points remain unclear, or if you have any further questions.

Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.