acceptodds
Withdrawn from ICLR 2027

PhysAg3D: Bridging 3D Generative and Agentic Models for High-Fidelity and Physically-Interactive Assets

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generating interactive 3D assets from a single image requires realistic appearance, fidelity to the depicted object, and plausible physical behavior. Yet detailed geometry alone does not determine how an object should move or deform. We introduce PhysAg3D, which couples 3D generative models with agentic reasoning and tool use. A vision-language agent first infers semantic parts, candidate interactions, and physical priors from the image; a 3D generator then produces a textured whole-object mesh from the same image. A hierarchical interaction representation links these revisable priors to mechanisms and the generated surfaces they control, organizing interaction construction and refinement. Guided by this representation and observations of the mesh, the agent uses geometric and physical tools to construct parts, joints, deformable regions, and surface bindings while retaining useful generated geometry and materials. Rendered interactions and execution measurements guide targeted revisions within a fixed budget. The framework accommodates both rigid articulation and compliant deformation. Quantitative and qualitative evaluations on real photographs and external benchmarks demonstrate detailed, image-aligned assets with plausible interactions. Ablations support bounded refinement and source-surface preservation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Interactive 3D Asset Genration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_SGuzOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes PhysAg3D, a framework for generating physically interactive 3D assets from a single image. The main idea is to combine a pretrained 3D generator, which generates a detailed textured 3D object, with a VLM-based agent that reasons about the semantic and physical structure of the object. The agent identifies how different parts should move or deform and uses geometry and simulation tools to construct parts, joints, deformable regions, and their connections to the visible surface. The generated interactions are then evaluated using visual and physical feedback, and the agent can revise the asset when the expected interaction is not achieved. The experiments show improvements over existing methods in both preserving the appearance of the input object and generating plausible interactions.

Critical strengths

The paper addresses an interesting and important problem of generating 3D assets that are not only visually similar to the input image but can also physically interact. Combining the detailed geometry and appearance from recent 3D generative models with agentic reasoning for articulation and deformation is interesting. I also like the idea of preserving the generated surfaces while adding physical interactions and using visual and physical feedback to revise the result. The paper has done decent evaluations, with comparisons on multiple benchmarks, a human study, and ablations, and the reported improvements over existing methods are good.

Critical weaknesses

My main concern is that the contribution appears to be primarily a system that combines several existing strong components rather than introducing a sufficiently novel technical method. The 3D object is generated using TRELLIS.2, a VLM is used for reasoning about the object and its interactions, and existing geometry and simulation tools are then used to construct and evaluate the physical asset. The hierarchical representation and iterative refinement organize these components, but I am not convinced that this provides enough technical novelty beyond the overall system design. I would like the authors to clarify what is non trivial in this pipeline that gives extra insights to the readers.

A second major concern is that the experiments do not sufficiently demonstrate the difficult multi-part setting that motivates the approach. TRELLIS.2 initially generates a whole-object mesh, after which the system needs to identify and construct the individual functional parts. Most examples shown in the paper contain relatively clear and externally visible parts, such as a lid, telescope joints, mixer head, excavator components, or camera lens.

I would like to see substantially more difficult examples such as furniture, lamps, or mechanical objects containing many small, nested, occluded, or completely hidden parts. In particular, it is unclear what happens when an important functional part is not represented in the initial generated geometry at all. Without such experiments, I am not convinced that the method generalizes beyond objects with relatively obvious functional decomposition.

Questions for authors

What is the main technical novelty beyond combining a 3D generator, VLM reasoning, geometry/physics tools, and an iterative agentic refinement loop?

How does the method handle objects containing many functional parts, particularly small, nested, or occluded components?

What happens when a functional part is completely hidden in the input image and consequently is not represented in the TRELLIS.2 reconstruction? Can the system actually generate the missing geometry?

Could you provide experiments on substantially more complex objects, such as furniture, lamps, or mechanical objects with many interacting components?

How much of the reported performance comes from the proposed hierarchical representation itself versus the underlying VLM, TRELLIS.2, and iterative tool-use pipeline?

Reviewer_z5tyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

PHYSAG3D combines pretrained 3D generation with VLM-based reasoning to create interactive assets. A hierarchical representation connects interaction goals, physical mechanisms, and asset implementations, guiding construction and feedback-based refinement. The contribution is primarily a system built around existing models rather than a newly trained generative model. Experiments report competitive appearance and interaction quality, with ablations supporting refinement and surface preservation.

Critical strengths
  1. Important and well-motivated problem. The paper addresses a key limitation of current 3D asset generation: lack of interactivity. It takes a systematic, agent-based approach that integrates existing models and leverages multimodal priors to guide the construction of interactive assets.
  2. Competitive reported results and appropriate distinctions between types of evidence.
Critical weaknesses
  1. Robustness to imperfect textures and materials is insufficiently explored. In practice, generated assets may contain missing or inaccurate textures and material information. These imperfections may affect downstream part identification, physical reasoning, and interaction construction.
Questions for authors

Ref to weakness

Reviewer_kjSeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how to generate 3D assets that combine detailed appearance with interactivity from a single image. The authors propose PhysAg3D, which uses TRELLIS.2 to generate a complete textured mesh. A vision-language agent then combines information from the input image, rendered views, and geometry, and invokes tools such as Blender and MuJoCo to construct movable parts, joints, or locally deformable regions and refine them based on feedback. The authors evaluate PhysAg3D against related methods on real-world objects with predefined actions, demonstrating the effectiveness of the proposed approach.

Critical strengths
  • The research addresses an important and meaningful problem and can possibly benefit robotics and related fields.
  • The method makes effective use of GPT-6-Astra’s capabilities in visual understanding, physical commonsense reasoning, tool use, and feedback-driven error correction.
  • The idea of using feedback from MuJoCo simulations to refine physical models is effective.
Critical weaknesses
  • Limited contribution: The method relies primarily on Astra’s capabilities, and the reflection-and-refinement workflow is already well established.
  • Unfair comparison: Agent-based baselines, such as ArtiCraft, should also use Astra as their underlying model. The current comparison does not clearly establish whether the performance gains stem from the architectural design or Astra’s stronger capabilities.
  • Insufficient output quality: The results still lack refinement, and their suitability for practical physics simulation remains unclear. For example, Figure 3 shows noticeable jagged artifacts at the junctions between the chair’s armrests and backrest. In Figure 5, the microwave’s interior structure revealed when the door opens is incorrect.
Questions for authors
  • Can the authors provide a controlled comparison that separates architectural improvements from Astra’s stronger capabilities?
  • How does the proposed system detect and correct jagged surfaces, component misalignment, and incorrect internal structures to ensure reliable physical simulation?
Reviewer_4dSxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PhysAg3D, a framework that couples TRELLIS.2 with a vision-language agent to generate 3D assets with high visual fidelity and physical interactivity from a single image. The core claimed contribution is a five-tuple hierarchical interaction representation (S,K,X,Γ,ρ)(S, K, X, \Gamma, \rho), built on a Blender and MuJoCo toolchain with at most two rounds of bounded repair under dual feedback from rendering and simulation. Experiments are conducted on a real-image test set, PhysXVerse, PhysX-Mobility, and PhysX-Bench.

Critical strengths

Well-defined problem with clear practical value: Decoupling high-quality whole-mesh generation from physical interaction construction precisely identifies the key pain point of end-to-end generative models, namely the loss of geometric high-frequency detail and material texture on complex articulated mechanisms.

Outstanding qualitative visual results: Examples such as the compass dial, camera body, and washing machine show markedly better surface completeness and texture preservation than the baselines.

Generally honest statement of experimental boundaries: The authors acknowledge that the ear cushion is an approximate compliant model, the telescope is kinematically rather than dynamically calibrated, and evaluation is video-based rather than simulator-log-based.

Critical weaknesses

The core geometric construction algorithm is under-specified and not independently reproducible: The paper provides only high-level descriptions of geometry editing and surface preservation. It does not specify how semantic parts are extracted from the complete closed mesh, how cutting boundaries are determined, how topological sealing is performed after cutting, how UVs and textures for newly created fracture surfaces are generated, or how geometric continuity is guaranteed after segmenting complex curved surfaces. These operations are central to the paper's claimed contribution, and the lack of concrete algorithmic detail constitutes a substantive technical gap.

The representational and methodological novelty is insufficiently validated: The mathematical formulation largely formalizes the organization of an engineering workflow rather than introducing a new computational method. More importantly, the necessity of the five-tuple representation is never demonstrated: Why cannot existing scene graphs or structured asset descriptions serve the same purpose? What concrete failures occur if Γ\Gamma or ρ\rho is removed? The ablation in Table 5 only compares One-shot, Full pipeline, and w/o surface preservation; it does not ablate the representation itself.

Experimental gains are confounded with the underlying generation backbone: PhysAg3D directly uses TRELLIS.2-4B with 1024_cascade and 2048-pixel textures, whereas the baselines (aDSL, ArtiCraft, Particulate) use older or lighter-weight generators. The large appearance-metric improvements (PSNR +2.7 dB over baselines, CLIP 0.892) therefore cannot be cleanly attributed to the proposed agentic framework. The Table 5 ablation only compares PhysAg3D's own configurations and does not resolve the cross-method backbone attribution problem.

Compliant deformation modeling is oversimplified relative to the claims: Equation (4), Vt=Vrest+W(Pt−Prest)V_t = V_{rest} + W(P_t - P_{rest}), is essentially a linear interpolation mapping, yet the paper does not explain how the physical nodes are modeled in MuJoCo, how constraints and material parameters are defined, how WW is computed, or how large deformations, stress distributions, anisotropy, or volume conservation would be handled. The authors themselves admit the ear cushion is only an approximate compliance model. Listing compliant deformation as a core contribution on par with rigid articulation constitutes overclaiming.

The experimental metrics do not fully substantiate the claimed physical-interaction capabilities: The main metrics (ACR, RAR, FVSP) are based on rendered video rather than simulator logs, and the authors concede in Appendix B that FVSP cannot demonstrate dynamic contact stability. The experiments mainly validate visually plausible motion rather than reliable physical interaction. Missing are dynamics-oriented validations such as physical parameter sensitivity, stability under varying payloads/contact conditions, and run-to-run consistency.

The system is insufficiently reproducible: The pipeline heavily depends on a commercial reasoning model (gpt-6-astra) and multi-tool orchestration, yet the paper provides no full system prompts, no tool API signatures, no end-to-end wall-clock time or token consumption, and no one-shot success rate. Appendix A.4 provides some configuration details (model versions, repair caps) but is far from sufficient for independent reproduction. Code absence alone is not grounds for rejection, but for such a black-box-dependent system, this lack of specification makes the core contribution largely unverifiable.

Questions for authors

Can you provide controlled same-backbone experiments that separate the contributions of the raw TRELLIS.2 mesh, the PhysAg3D-constructed mesh, and alternative construction strategies applied to identical input meshes?

Please specify in detail the mesh-partitioning algorithm: How are cutting planes determined? How are the resulting cavities sealed topologically? How are UVs and textures for newly created faces generated?

What unique capability does the five-tuple representation provide over scene graphs or JSON-style state records? What concrete failure modes arise if Γ\Gamma or ρ\rho is removed?

How are the physical nodes in Equation (4) modeled and driven in MuJoCo? How is WW computed? Can this mechanism generalize to large-deformation soft bodies?

Can you provide simulator-log-based dynamic metrics (contact forces, joint torques, stability over repeated runs) to support the physical-interaction claims?

What is the average wall-clock time for a complete two-round repair, and what fraction of the 50 real objects passed with a one-shot run? During the rebuttal, can you provide sanitized prompt templates, tool API documentation, and a code repository link?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.