acceptodds
Under review as a conference paper at ICLR 2027

Towards Physically Grounded Articulated 3D Assets: A Multi-Agent Pipeline and Benchmark

Abstract

Recent advances in 3D generation have significantly improved aesthetic quality, but they do not ensure that objects behave correctly under physical interaction. In this paper, we study physically grounded articulated 3D asset generation—moving the goal from reproducing how objects look to capturing how they move and function. Inspired by the literature on procedural and agentic generation, we present Artisan, a system that extends the latest method Articraft by introducing a multi-agent pipeline with physical simulation feedback, enabling comprehensive attributes beyond geometry, appearance, and physically plausible articulation, such as collision-free motion, stability under gravity, and accurate mass and material properties. Furthermore, we construct a unified benchmark spanning four key aspects: spatial fidelity, physical plausibility, downstream task performance, and overall quality. The benchmark encompasses 100 reference assets from ArtVIP and LightWheel, 50 stability prompts, and 50 ABO objects with reference masses, with protocols evaluating various metrics. Through extensive evaluation of more than 10 representative models, Artisan achieves competitive spatial fidelity while attaining the lowest collision rates and highest stability rates among the evaluated methods. Together, our system and benchmark establish insights for advancing 3D generation towards physically plausible, functionally usable assets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
articulated 3D asset generation, LLM agents, physical simulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_r8EsOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents ARTISAN, a multi-agent framework for generating articulated 3D objects with detailed attributes, including mass, physical parameters such as roughness and metallicity, and material properties. In this system, the design agent generates a design blueprint from text or image inputs, the coding agents write code for the asset based on the blueprint, and the reviewer agent evaluates the generated asset and provides feedback. The generated assets improve upon existing methods in terms of both visual appearance and the quality of physical attributes.

Critical strengths
  • The paper proposes a multi-agent system for generating articulated 3D objects. The task itself is important, and the proposed method is interesting, particularly the use of a design agent to generate a blueprint from the inputs.

  • The robotics experiments strengthen the paper, although their scope is somewhat limited.

Critical weaknesses
  1. The scope of the robotics experiments is somewhat limited. The experiments demonstrate that combining native assets with ARTISAN-generated assets improves the policy success rate, but this also increases the total number of assets used. In addition, no real-world policy evaluation is conducted. Thus, the authors take a step forward in real-to-sim but do not address real-to-sim-to-real.

  2. Stability does not necessarily imply greater realism. The agent could assign a higher density to the base of an object and a lower density to its upper parts to achieve stability. However, the authors report only the mass difference at the whole-object level. Therefore, there is no evidence that the generated assets have realistic mass distributions across their parts. Physical parameters such as friction and damping are also not evaluated. The robotics experiments could serve as an indirect assessment, but their scope is limited, as discussed above.

  3. Minor suggestions: The reference formatting is inconsistent: some entries use full names, while others use abbreviations; some include page and volume information, while others omit it.

Overall, I lean toward acceptance and would be happy to (and probably will) raise my score further after reading the other reviewers' comments and the authors' rebuttal.

Questions for authors

My concerns are outlined in the weaknesses section. I have no additional questions.

3 comments
AuthorsOpenReview ↗

Response to Reviewer r8Es

We thank the reviewer for recognizing the importance of articulated-asset generation, the value of blueprint-based design, and the contribution of our robotics experiments, and for the suggestions on data augmentation and physical attributes.

[W1]: The scope of the robotics experiments is somewhat limited

(a) The gain does not come from more assets alone. High-quality articulated training assets are scarce and costly to expand. Artisan addresses this bottleneck by generating additional usable assets, enabling better generalization to unseen native objects. Table 3(a) reports an increase from 80% to 84% on RoboTwin and from 39% to 43% on RoboCasa when Artisan assets augment the native training set.

With equal numbers of generated candidates, Artisan achieves usable-asset rates of 100% on RoboTwin and 44% on RoboCasa, compared with 62% and 8%, respectively, for Articraft. Under this equal candidate-generation scale, Artisan-only policies achieve higher average success than Articraft-only policies:

Training assetsRoboTwin success ↑RoboCasa success ↑
Articraft only60%5%
Artisan only63%27%

We additionally compare Native + Articraft with Native + Artisan using equal numbers of generated candidates and the same training and held-out evaluation protocol:

Training assetsRoboTwin success ↑RoboCasa success ↑
Native + Articraft81%36%
Native + Artisan84%43%

Artisan augmentation achieves higher success on both benchmarks, supporting its greater utility as a source of additional training assets.

(b) Real-world evaluation. We agree that sim-to-real policy transfer is not evaluated. Our contribution focuses on real-to-sim, generating articulated assets from real-world inputs that are physically valid and ready for simulation. Transferring policies back to the real world depends largely on factors outside the asset-generation scope of this paper.

We see a full real-to-sim-to-real evaluation as an important next step and have discussed it in the limitations section.


[W2]: Stability does not necessarily imply greater realism

The absence of measured part-level mass ground truth is a limitation of the current evaluation. ABO provides whole-object masses, while the articulated-asset datasets used here also do not provide experimentally verified part-wise masses. We identify this collection and release as future work.

To test whether our stability advantage depends on predicted mass allocation, we conduct an additional experiment on all 50 Stability objects. We remove the predicted part masses and inertias and recompute them using a uniform density of 1000 kg/m³, while keeping geometry, articulation, and other parameters unchanged:

ConfigurationRest stability ↑Open stability ↑
Ours(GPT-6 Astra)98%96%
Ours(GPT-6 Astra) with uniform density92%92%
Articraft (GPT-6 Astra)82%82%

Even without predicted part masses and inertias, Artisan exceeds Articraft's reported stability by 10% in both states. This supports the structural stability of our generated assets.


[W3]: Reference formatting

We have standardized conference and journal names throughout the bibliography and completed the available page and volume information, including the missing page ranges for ABO and TRELLIS and the PMLR volumes for BEHAVIOR-1K and image2mass.

Thanks for the authors rebuttal! I don't have more questions for now and I believe this paper provides useful insights to the community. I'll decide whether to raise my score based on other reviewers and AC's feedback.

Thank you for your encouraging follow-up and for recognizing the value of our work. We are glad that our rebuttal has addressed all of your concerns.

We have also provided additional experiments and clarifications in response to the other reviewers. As you revisit the discussion, we would be grateful if you would consider whether this additional evidence supports a higher rating.

Thank you again for your time and thoughtful review.

Reviewer_gfEbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduced a multi-agent articulated object generation named Artisan. It involves three agents, i.e. a design agent, a coding agent and a review agent Given a text prompt and (optionally a reference image), the agents work together towards generating a high-fidelity and physically plausible 3D asset in simulation. The paper also introduced a unified benchmark evaluating four key aspects of generated assets.

Critical strengths

The idea of introducing "reusable standard connectors for joint connections" to agents is novel and interesting for generating articulated objects. It offers insights on how to effectively generate articulated objects in a bottom-up manner.

Critical weaknesses
  1. My main concern is on the paper's proposed benchmark, or more specifically the metrics. It seems that the "spatial fidelity" and "physical properties" both requires a "ground-truth or reference object", which means that they cannot be used to evaluate the quality of objects that are "imagined". On the other hand, one might think that "if such ground-truth objects did exist, there is no need for object generation", which I believe should be properly discussed in the paper. Also, regarding "robotic task performance", I find it difficult to understand the logic for "higher downstream task performance -> higher asset quality".
  2. The content arrangement of the paper could be improved. For example, in Line 210-213, the authors introduced three components that appears to be parallel, whereas the first is detailed in the appendix, the second is right under that sub-section, and the third is in another sub-section.
Questions for authors
  1. Please refer to weakness 1.
  2. In the introduced pipeline, it seems that the design agent does a one-time job before the coding agent and the review agent begins to cooperate. What happens if the design agent makes a mistake? Are there procedures or mechanisms that prevent such issues?
4 comments
AuthorsOpenReview ↗

Response to Reviewer gfEb

We thank the reviewer for recognizing the novelty of our reusable joint connectors and their value for bottom-up articulated-object generation, and for the suggestions on evaluation and design verification.

[W1 & Q1]: Reference-based evaluation and downstream asset quality

(a) Several of our metrics need no reference, so they apply to imagined objects. Stability, downstream policy success and user study are evaluated without any ground-truth object. Collision can also be evaluated without a reference, as long as the intended motion of each articulated part is known. These metrics therefore apply directly to imagined assets.

(b) Evaluating against ground truth is the stricter and more meaningful test. The ground-truth objects serve only as a held-out test set, where Artisan never sees them as input. The reference set is what lets us verify objectively whether a generator recovers correct geometry, articulation and physical properties. But generation remains necessary for building large-scale simulation-ready articulated assets. Besides, this is standard practice for evaluating conditional generation. A benchmark with references is needed to measure fairly how well a model generates from a given condition, as in image-to-3D (GSO, Objaverse), image-to-articulated-3D (PartNet-Mobility, LightWheel), image-to-video (Physics-IQ) and novel view generation benchmarks (Syn4D).

We also believe this is more reliable than evaluation by VLMs, which many existing works use instead. VLM judges cannot measure geometry or physical parameters accurately, and their scores cannot be reproduced across model versions.

(c) Why higher downstream success indicates higher asset quality. We do not claim that downstream success captures every aspect of quality. It measures one aspect directly: how useful the assets are as training data. This follows the standard way of evaluating synthetic data, in which a model is trained on generated data and tested on real data, with the generator as the only variable.

Overall, our evaluation covers four dimensions: fidelity (reference-based, following prior image-to-articulated-3D work), physical validity (reference-based & reference-free), and downstream utility (reference-free), along with user study (reference-free).


[W2]: Organization of the method description

Thank you for the suggestion. We have revised Section 3.2 to present inspection tools, standard connectors, and physics feedback in a consistent order, briefly explaining each component.


[Q2]: Design self-check and correction

Design errors are guarded against at two stages: (1) Self-verification within Design (Sec. 3.1, Blueprint Verification and Visual Reference). Before producing the design board, the Design Agent checks its blueprint against the input and revises inconsistent parts, motions, and functions. Appendix F.1, Figure 16, gives a concrete example: the stand-mixer agent writes the blueprint at turn 1, checks it at turn 2, corrects the accessory-cap location at turn 3, and verifies the revision at turn 4. Design therefore includes correction before coding begins. (2) The independent Review Agent will subsequently check the implementation against the original prompt. Hence, if a design error that gets past the self-check still appears as a mismatch, the Review Agent states the error explicitly and asks the Coding Agent to correct it.

I thank the authors for the response. Some of my concerns are addressed, yet I do not think it gives me enough support to increase my rating to 3 (I would choose a 2.5 if there was such an option). My concerns remain on the proposed benchmark: it is listed as one of the key contributions of the paper, yet it only fully functions when paired with a set of objects with ground truth mesh, which also lacks in scale as a dataset.

AuthorsOpenReview ↗

Response to Reviewer gfEb

Thank you for clarifying your remaining concerns. We believe that establishing a reference-based benchmark for physical-attribute accuracy is itself an important contribution to the community.

First, reference-based evaluation on curated, modest-sized test sets is established practice in image-to-3D and articulated-object modeling. Representative examples include:

  • LRM [1] evaluates on 100 GSO objects [2], comparing generated appearance and geometry against references.
  • CRM [3] evaluates on 30 GSO objects [2] with ground-truth meshes and rendered views.
  • InstantMesh [4] evaluates on 300 GSO objects [2] and 130 OmniObject3D objects [5].
  • SINGAPO [6] uses 77 PartNet-Mobility test objects [7] and 135 ACD objects [8], evaluating part geometry and articulated structure against ground truth.
  • Particulate [9] introduces a benchmark of 243 Lightwheel objects [10] with articulation annotations.

Our benchmark comprises 200 test cases: 50 ArtVIP objects, 50 Lightwheel objects, 50 ABO objects for mass evaluation, and 50 Stability cases. This places our evaluation in the same tens-to-hundreds scale as these prior studies. Both reference-based evaluation and curated test sets at this scale are established practice. Our benchmark extends this practice to physical-attribute accuracy and simulator readiness.

Second, ground truth provides an objective basis for precise, quantitative evaluation. We recognize reference-free assessments, including human evaluation in SceneCraft [11] and Holodeck [12], and LLM-based judging in BlenderLLM's CADBench [13]. Other approaches include visual and geometric similarity in BlenderGym [14] and visual judging combined with scene inspection in BlenderWorld [15]. However, comprehensive evaluation of physical attributes against trustworthy ground truth remains limited. Without reference data, we cannot reliably quantify how closely generated assets match real-world properties. Preference and rubric scores do not directly measure these deviations. Our benchmark enables such measurements under a common, reproducible protocol, supporting credible and fair comparisons. Establishing this reference-based evaluation is itself a contribution to the community. Although reliably annotated assets remain limited, the evaluation protocols are general, and we can expand the benchmark as more such assets become publicly available.

References

[1]: Yicong Hong et al. LRM: Large Reconstruction Model for Single Image to 3D. ICLR, 2024.

[2]: Laura Downs et al. Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items. ICRA, 2022.

[3]: Zhengyi Wang et al. CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model. ECCV, 2024.

[4]: Jiale Xu et al. InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models. arXiv:2404.07191, 2024.

[5]: Tong Wu et al. OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation. CVPR, 2023.

[6]: Jiayi Liu et al. SINGAPO: Single Image Controlled Generation of Articulated Parts in Objects. ICLR, 2025.

[7]: Fanbo Xiang et al. SAPIEN: A SimulAted Part-Based Interactive ENvironment. CVPR, 2020.

[8]: Denys Iliash et al. S2O: Static to Openable Enhancement for Articulated 3D Objects. WACV, 2026.

[9]: Ruining Li et al. Particulate: Feed-Forward 3D Object Articulation. CVPR, 2026.

[10]: Lightwheel. Lightwheel SimReady Asset Library. Official repository: LightwheelAI/Lightwheel-simready-asset.

[11]: Ziniu Hu et al. SceneCraft: An LLM Agent for Synthesizing 3D Scenes as Blender Code. ICML, 2024.

[12]: Yue Yang et al. Holodeck: Language Guided Generation of 3D Embodied AI Environments. CVPR, 2024.

[13]: Yuhao Du et al. BlenderLLM: Training Large Language Models for Computer-Aided Design with Self-improvement. arXiv:2412.14203, 2024.

[14]: Yunqi Gu et al. BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing. CVPR, 2025.

[15]: Xinyu Fang et al. BlenderWorld: Benchmarking Visual Agents for 3D Creation from Real-World References. arXiv:2610.23986, 2026.

Thank you for your follow-up and for acknowledging that some of your concerns have been addressed. In our latest response, we further clarified the role of ground-truth references and compared our benchmark’s evaluation scale with established image-to-3D and articulated-object benchmarks.

We would appreciate your feedback on whether these clarifications address your remaining concerns. If so, we hope you would reconsider your rating.

Thank you again for your time and constructive discussion.

Reviewer_VW1HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Artisan, a multi-agent pipeline for generating physically consistent articulated 3D assets. It builds on the Articraft pipeline, adding multiple agents and physical simulator feedback. The physical simulator is used in an iterative process to reduce collisions under articulation of the asset and encourage it to be stable under gravity. The paper also presents an evaluation benchmark to measure spatial fidelity, physical attributes, downstream tasks, and human-judged quality. The assets produced by Artisan display much lower collision rates and much higher stability compared to other methods.

Critical strengths
  1. Physical validation substantially improves the articulation quality of the objects. Compared to alternate methods, Artisan achieves much lower collision rates and much better stability.
  2. The benchmark gives a thorough evaluation. The benchmark includes objective metrics regarding collisions and visual properties. It also considers impact on downstream tasks and human judgement. This provides a thorough analysis of asset quality.
Critical weaknesses
  1. Artisan is substantially more expensive than the baselines, making it difficult to establish the benefit of the proposed architecture. The cheapest Artisan method is \5.79perasset,comparedtothemostexpensivealternatemethodat5.79 per asset, compared to the most expensive alternate method at \\1.32 per asset (Table 13). As such, it is unclear if the improvements in asset quality arise from Artisan’s multi-agent pipeline or simply more computation. A compute-matched budget comparing Artisan to Articraft or Vanilla codex would be much more convincing. Additionally, if given programmatic access to a physics simulator, could Codex achieve similar results?
  2. The downstream usefulness of Artisan’s assets is not well established. While reduced collisions and stability are good properties, it is not clear how important these are for improved downstream performance in robotics or other domains. In Table 3a, the downstream success rate does not clearly demonstrate Artisan’s advantage. Compared to the native assets, adding Artisan produces minimal improvement. The comparison to “Articraft + native” is also not included
Questions for authors
  1. How would Artisan compare to Articraft or Codex with an equivalent computation budget? Would it be possible to provide an ablation on the multi-agent architecture by giving codex (or another method) access to simulator feedback?
  2. In Table 3a, how would “Native + Articraft” compare to “Native + Artisan”? Is there a reason this comparison was omitted?
  3. In Table 3a, are the differences between “Native + Artisan” and “Native” statistically significant?
  4. Can the authors clarify the criterion on lines 308 and 309? How does the “maximum tilt angle” differ from “rotation change” and why is the threshold much higher?
3 comments
AuthorsOpenReview ↗

Response to Reviewer VW1H [1/2]

We thank the reviewer for recognizing the benefits of physical validation and for the suggestions on computation and downstream evaluation.

[W1 & Q1]: Computation budget and simulator access

In summary, we address this issue in three parts: (a) baselines are not limited by budget; (b) a cheaper Artisan configuration with comparable or even better quality; (c) an ablation that gives Codex physical simulator feedback. We discuss them in detail below.

(a) Baselines are not budget-limited. The costs in Table 13 are actual spend, not the budget caps assigned to the baselines. In particular, all baseline runs completed their generation workflows without budget truncation. Their lower spend reflects that they stopped sooner, not that they were cut off. Increasing the budget ceiling does not make them continue computation after finishing. A larger allowance does not itself introduce additional compute into a baseline.

(b) Cheaper Artisan configuration. Thanks to our multi-agent architecture, we can use a Mixed configuration that retains GPT-6-Astra for Design and uses GPT-6.1-Sol for Coding and Review, all at high reasoning effort to reduce generation cost. This cuts cost by 59% ($5.79 → $2.38 per asset). Quality stays the same or improves: 2D IoU, PSNR, CLIP and all three collision rates get better, and the remaining metrics change only slightly. A single-agent pipeline like Codex cannot assign models by role in this way.

(c) Ablation: Codex with simulator feedback.

As suggested, we equip Codex with our pipeline's simulator feedback and evaluate it under the paper's protocol using GPT-6-Astra/high. The results are:

Method2D IoU ↑PSNR ↑CLIP ↑Rest oCD ↓Rest pIoU ↑Open oCD ↓Open pIoU ↑Collision: Single / Double / Multiple ↓Stability: Rest / Open ↑Mass APE ↓Cost (USD/asset) ↓
Ours(Astra)0.767518.650.92390.00110.35870.00210.34171.7% / 3.7% / 13.0%98% / 96%0.260$5.79
Ours(Mixed)0.777218.720.92970.00150.35380.00240.34471.4% / 3.2% / 10.5%96% / 96%0.271$2.38
Articraft (Astra)0.760218.620.91970.00220.32440.00340.322445.5% / 60.7% / 78.8%82% / 82%—$0.95
Codex (Astra)0.750418.160.91690.00320.33310.00390.309532.4% / 47.0% / 63.7%86% / 82%0.303$1.07
Codex (Astra) + Simulator0.733418.240.92100.00160.32700.00500.31013.0% / 6.3% / 14.4%96% / 96%0.284$1.25

As shown in the table, our simulator feedback substantially improves Codex's collision and stability performance, suggesting its value beyond our own pipeline. But Ours(Astra) still retains lower collision rates and higher reconstruction fidelity: appearance IoU is 0.7675 versus 0.7334, and closed/open pIoU is 0.3587/0.3417 versus 0.3270/0.3101. These results support the benefit of our multi-agent architecture.

AuthorsOpenReview ↗

Response to Reviewer VW1H [2/2]

[W2 & Q2]: Downstream utility and Native + Articraft Evaluation

Collision-free articulation and stability are prerequisites for usable robotic training data. Assets with invalid collision geometry or insufficient stability cannot reliably produce successful demonstrations for policy training, directly limiting the usable training data supplied by the generator. This is reflected in Section 5.2, Table 3(b): 100% versus 62% of Artisan and Articraft assets support demonstration collection on RoboTwin, and 44% versus 8% on RoboCasa. The failure analyses in Appendices E.1–E.2 identify incorrect collision geometry, interpenetration, and instability as concrete causes of collection failure. Physical validation therefore addresses a practical bottleneck in obtaining training data, rather than improving simulation metrics alone.

Furthermore, collision-free articulation and stability also matter for real-world functional design. As agentic generation will move toward assisting engineers in designing physical products (e.g., furniture, appliances, mechanisms), a generated design is useful only if its parts move without interpenetration and the object remains stable in use. By enforcing these properties during generation, Artisan takes a step toward functional 3D design beyond robot training. We have added this perspective to the discussion of future work.

The benefit also reaches the downstream policy: under the same candidate-generation scale, Artisan-only training achieves 63% versus 60% task success on RoboTwin and 27% versus 5% on RoboCasa (Table 3(a)). Together, these results connect physical usability to both the availability of successful demonstrations and generalization to held-out objects.

We have run the missing Native + Articraft setting, using the same protocol as Table 3(a): same policy, same training and evaluation protocol on the same held-out objects:

Training assetsRoboTwin success ↑RoboCasa success ↑
Native80%39%
Native + Articraft81%36%
Native + Artisan84%43%

Native + Artisan outperforms Native + Articraft by 3% on RoboTwin and 7% on RoboCasa. Artisan augmentation improves average success over Native on both benchmarks, whereas Articraft augmentation provides a smaller gain on RoboTwin and reduces success on RoboCasa. This comparison supports the downstream utility of Artisan's generated assets as additional training data.

We have added these results to the revised PDF.


[Q3]: Statistical uncertainty of the downstream gains

In short, the gain is significant on RoboCasa and not on RoboTwin, where the evaluation set is small.

Evaluation setup. Each policy is evaluated over 50 rollouts per held-out test asset, totaling 250 rollouts across the three RoboTwin tasks and 3,150 across the eight RoboCasa tasks; this repeated evaluation reduces sampling variability.

Results and analysis. Native + Artisan increases average success from 80% to 84% on RoboTwin and from 39% to 43% on RoboCasa. We also assess statistical significance: the 95% confidence intervals for the gains are [−3.51, 10.84]% on RoboTwin (p = 0.317) and [1.76, 5.99]% on RoboCasa (p < 0.001). The RoboCasa results support a statistically significant improvement under this analysis. Both benchmarks show positive average gains.


[Q4]: clarify the criterion on Maximum Tilt and Rotation Change

The two criteria measure different quantities, so their thresholds are not directly comparable.

Maximum tilt measures the largest angular deviation from the initial upright direction and detects tipping. Rotation change measures residual orientation drift after settling and checks whether the object remains stationary. Therefore, 30° permits moderate leaning without tipping, whereas 0.01 rad requires negligible residual rotation. We have clarified this distinction in Section 4.1.

Thank you for your valuable feedback. Following your suggestions, we have added the Codex + Simulator and Native + Articraft comparisons, evaluated a lower-cost model configuration, and provided statistical analysis and clarifications on the evaluation criteria.

We would greatly appreciate it if you could revisit our responses and let us know whether any concerns remain. If these additions have addressed your concerns, we hope you would consider raising your rating.

Once again, we sincerely thank you for your time, effort, and thoughtful review of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.