acceptodds
Withdrawn from ICLR 2027

Phi-Art: Physics-Aware Articulated Object Generation from a Single Image with a Multimodal Large Language Model

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generating articulated assets from a single image requires more than reconstructing visible surfaces. Part geometry, kinematic structure, and physical attributes must be inferred from incomplete visual evidence to support interactive simulation. How- ever, existing models are struggling to tackle these problem all together. Thus, we introduce Phi-Art: -Art, a multimodal framework that couples structured object prediction with parallel geometry decoding. A multimodal language model predicts part semantics, 3D bounding boxes, joint parameters, and physical attributes, providing a coarse spatial plan for asset generation. A Qformer-style decoder combines the model’s hidden representations with image features to predict whole-object geometry codes in parallel, separating geometric decoding from autoregressive text generation. We further corrupt structured descriptions during training to encourage the use of visual evidence and reduce reliance on accurate textual layouts. The predicted geometry and layouts guide part synthesis, followed by kinematic assembly and export to simulator formats. Experiments on PhysXNet and Mobility show that our method outperforms the existing baselines in the geometry accuracy, training and inference efficiency with a competitive physical attributes prediction. We will release our code, model checkpoints, and processed training data

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Simulation-ready Asset Generation, 3D Generation, Robot Simulation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_EBt4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Phi-Art generates articulated 3D assets from a single image by fine-tuning a multimodal language model to predict part layouts, kinematic structures, physical attributes, and coarse whole-object geometry. A frozen OmniPart model then generates textured part meshes, which are assembled into URDF/MJCF assets. Experiments evaluate reconstruction quality, structural and physical predictions, and qualitative robotic interactions in simulation.

Critical strengths
  1. The framework produces textured articulated assets with part structure, joint parameters, and physical attributes from a single image, with URDF/MJCF export for simulation.

  2. Evaluation on 1,310 objects demonstrates improved geometric reconstruction and part localization, supported by training-data and compute-scaling studies.

Critical weaknesses
  1. Limited methodological novelty and insufficient justification. The method combines a fine-tuned ShapeLLM, parallel coarse-geometry prediction, and frozen OmniPart synthesis. The incremental contribution remains insufficiently established, particularly without a controlled comparison against pretrained or fine-tuned TRELLIS using identical part layouts, masks, and downstream synthesis.

  2. Incomplete comparisons with recent methods. Comparisons with PWM-ArtGen [1], PhysX-Omni [2], and image-conditioned Articraft [3] are missing. ArtLLM [6], SIMART [5], and Particulate [7] could provide additional baselines with a common image-to-3D frontend. Sketch2Arti [4] and LAM [8] also warrant discussion, while acknowledging their different input requirements.

  3. Limited gains in articulation and physical prediction. In Table 2, Phi-Art’s kinematic score is lower than those of PAct and PhysX-Anything (0.68 versus 0.73 and 0.74), and drops to 0.62 with automatic masks. Material, affordance, and description scores also remain below PhysX-Anything, leaving the benefits beyond geometry and scale unclear.

  4. Insufficient quantitative validation of physical usability. VLM-based motion scores and rendered property-map PSNR do not directly establish accurate joint parameters or reliable physical behavior. The simulation examples are qualitative; quantitative evaluations of joint accuracy, interpenetration during motion, and manipulation success are missing.

  5. Ablations do not sufficiently isolate the proposed contributions. Removing the Q-Former also switches geometry generation from parallel to autoregressive prediction, confounding architecture and decoding strategy. A matched image-only geometry decoder without VLM conditioning would help establish whether the proposed semantic–geometric coupling is necessary.

References

[1] PWM-ArtGen: Part World Model for Articulated Object Generation.

[2] PhysX-Omni: Unified Simulation-Ready Physical 3D Generation for Rigid, Deformable, and Articulated Objects.

[3] Articraft: An Agentic System for Scalable Articulated 3D Asset Generation.

[4] Sketch2Arti: Sketch-based Articulation Modeling of CAD Objects.

[5] SIMART: Decomposing Monolithic Meshes into Sim-ready Articulated Assets via MLLM.

[6] ArtLLM: Generating Articulated Assets via 3D LLM.

[7] Particulate: Feed-Forward 3D Object Articulation.

[8] LAM: Language Articulated Object Modelers.

Questions for authors
  1. What is the key methodological advance beyond combining ShapeLLM, parallel geometry decoding, and OmniPart? Would pretrained or fine-tuned TRELLIS achieve comparable results under the same part layouts, masks, downstream synthesis, and training budget?

  2. Could you compare with PWM-ArtGen, PhysX-Omni, and image-conditioned Articraft, and consider ArtLLM, SIMART, and Particulate with a common image-to-3D frontend? Please also clarify the relationship to Sketch2Arti and LAM, acknowledging their different input requirements.

  3. How should we interpret the lower kinematic score than PAct and PhysX-Anything, and the lower material, affordance, and description scores than PhysX-Anything? What evidence establishes benefits in articulation and physical prediction beyond improved geometry and scale?

  4. Could you provide quantitative evaluations of joint types, axes, positions, and motion limits, together with interpenetration rates or manipulation success rates? Which predicted physical attributes actually affect the simulated behavior, and how is their accuracy validated?

  5. Could you disentangle the effects of the Q-Former architecture and parallel decoding in the ablations? In particular, how does a matched image-only geometry decoder without VLM conditioning perform, and does semantic conditioning provide a measurable benefit?

Reviewer_7JkTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Phi-Art, a framework that incorporates both VLM and dedicated 3D generator to create physics-aware articulated object generation.

Critical strengths

None

Critical weaknesses

Unfortunately, I find myself unable to follow the text in the method section, there seems to be a lot of concepts or notations that are brought up without proper introduction or explanation. Below I give some examples:

  1. In Fig.2, Round 1, it seems that the 3D MLLM is giving a "Visible Part Seg", but this is not mentioned in Sec. 3.1
  2. In L188, the authors discussed about a geometry token, i.e. <mesh_out>, how is this token acquired? Also, I only found three references to "<mesh_out>" throughout the paper, two in Fig.2, one in L188; where and how is this token used?
  3. Section 3.2 mentioned terms like "conditioning queries" and "geometry queries", what's the motivation behind them?
  4. Section 3.3 mentioned training, but the authors did not seem to clearly mention which models are being trained.
Questions for authors

Please refer to weaknesses, I believe the method section requires a full clarification. I am still willing to re-evaluate the paper with proper clarifications.

1 comment

We thank the reviewer for the careful reading and constructive feedback. We apologize for the confusion and provide the following clarifications.

1. Visible-part segmentation (Fig. 2, Round 1). The VLM does not directly predict pixel-level segmentation masks. Instead, it predicts 2D bounding boxes of visible object parts using 128-bin coordinate quantization. These boxes are then used as prompts for SAM3 to obtain pixel-level part masks, which condition the downstream OmniPart generation stage.

2. The <mesh-out> token (Sec. 3.1). <mesh-out> is a registered special token that serves as an anchor for extracting a geometry-conditioning representation from the VLM. Specifically, we extract its associated final-layer hidden state (3584 dimensions) and project it to 1024 dimensions. This representation is used as conditioning memory in the subsequent fusion decoder. Its representation is indirectly optimized through geometry-code prediction supervision.

3. Conditioning queries and geometry queries (Sec. 3.2). These two query groups serve different purposes. The 128 conditioning queries aggregate semantic and structural information from the VLM, while 64 DINO queries aggregate visual evidence from the input image. A fusion decoder integrates these representations, with the conditioning queries attending to 193 memory tokens (128 conditioning tokens, one <mesh-out> representation, and 64 DINO tokens). Then, the 1024 learned queries, equipped with 3D positional embeddings, are used by geometry decoder to predict the fixed-length sequence of 1024 VQ-VAE codes in parallel. This separation decouples multimodal condition aggregation from detailed geometry decoding.

4. Trainable and frozen components (Sec. 3.3). We fully fine-tune the VLM language transformer and train its multimodal merger/projector and the Q-Former. The VLM visual patch embedding and visual transformer remain frozen. DINOv3 features are precomputed, and DINOv3, the VQ-VAE codebook and decoder, OmniPart, and SAM3 are all frozen. The two VLM rounds share the same parameters and training pipeline. Language loss supervises the response tokens, while geometry losses propagate through the geometry-conditioning pathway to the trainable components.

We appreciate the reviewer for identifying these gaps in the method description. We hope these clarifications make the architecture and its implementation easier to follow.

Reviewer_nYD5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the generation of 3D assets with part structure, joints, and physical properties from a single image. The method first uses a VLM initialized from ShapeLLM-Omni to predict a structured object description. The hidden representation from a second query is then fused with DINOv3 image features to predict geometry codes in parallel. A frozen VQ-VAE decodes these codes into an occupancy representation, which is passed to a frozen OmniPart model together with the input image, part bounding boxes, and segmentation masks to generate textured part meshes. These meshes are then assembled according to the predicted joint relationships and exported in URDF/MJCF format.

Critical strengths
  1. The authors fuse the VLM’s hidden representation with DINOv3 image features to predict coarse geometry codes in parallel, and use structured-context corruption and silhouette supervision to reduce reliance on textual layouts. The design is well motivated.

  2. The method improves geometric reconstruction quality while also reducing inference time. Compared with PhysX-Anything, end-to-end inference time decreases from 212 seconds to approximately 176 seconds.

Critical weaknesses
  1. The consistency between the predicted joint parameters and the generated geometry has not been directly validated. The paper shows improvements in geometric reconstruction in the rest pose, but its kinematic evaluation relies mainly on VLM-based scores. It reports neither direct joint parameter errors nor geometric reconstruction performance across different joint configurations. The kinematic score in Table 2 also does not exceed that of the main baseline.

  2. The accuracy of the physical properties and their role in simulation remain insufficiently validated. Material, affordance, and description are evaluated mainly using PSNR on rendered attribute maps. These scores depend on both geometry and attribute assignment, making it difficult to assess attribute prediction accuracy in isolation. For numerical properties such as density and Young’s modulus, the paper does not report direct errors against the annotated values. The paper does not explain how contact friction, joint damping, and collision settings are determined. It is therefore unclear whether the interaction demonstrations require additional manual configuration. The selected frames in Figure 7 are also insufficient to assess the full contact process or check for interpenetration.

Questions for authors

see weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.