acceptodds
Under review as a conference paper at ICLR 2027

CRAFT: ContRollable Generation of Articulated Objects From Image and Text

Abstract

Controllable articulated-object generation is important for building interactive 3D worlds, supporting embodied intelligence and robotic simulation, and creating functional digital assets. A reference image reveals what an object looks like, but not necessarily how a user wants its parts to move. We introduce CRAFT, a framework for visually grounded, controllable articulated-object generation from a reference image and a textual articulation specification. CRAFT separates coarse structural control from detailed asset synthesis through an editable articulated 3D proxy that represents part dimensions and positions, spatial relationships, and kinematic parameters. We first construct an initial proxy using a category-informed structural initialization strategy. We then refine its spatial arrangement through adaptive structural optimization that combines instance-conditioned geometric objectives with semantic guidance from an image-derived 3D estimate. Finally, we generate detailed part assets and assemble them according to the refined proxy. Experiments demonstrate that CRAFT generates articulated objects consistent with the reference appearance while enabling text-specified control over part structure and motion.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Articulated Object Generation, Part-Aware 3D Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dXMgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CRAFT, a framework for articulated object generation based on a reference image and a text prompt. The reference image provides appearance control, whereas text description primarily focuses on articulation. More specifically, the proposed approach repeatedly utilizes SAM3D to provide physical dimensions of the whole object as well as its parts, and assembles them together based on a structural configuration acquired via optimization.

Critical strengths
  1. The proposed topic of regulating the generated articulated object with human language is indeed meaningful.
  2. The paper is quite clearly written and easy to follow.
Critical weaknesses
  1. The proposed method appears slightly too complex for a pipeline, making it easier to break during generation. For example, in Eq.8, the authors formulated the object structure as an optimization problem with 6 types of losses. Without proper weight tuning, the optimization may not progress as expected.
  2. The proposed method may not be able to expand to more complex (rigid) objects due to its dependency on bounding boxes. In Sec 3.3.2, the authors mentioned penalizing bounding box overlaps, yet in reality, overlapping bounding boxes does not necessarily indicate overlapping parts.
Questions for authors
  1. In Eq.8, are the three "w"s manually or automatically selected? If the latter, how is it determined?
  2. In Fig.4 bottom left, I noticed the second generated cabinet (third picture) has door handles facing the wrong direction, is there a way to avoid this?
2 comments

We thank the reviewer for the positive comments on the topic and the writing.

W1. The optimization requires no weight tuning:

  • The three mandatory terms (LolL_{ol}, LregL_{reg}, LSAML_{SAM}) always have unit weight.
  • The three optional terms have binary weights w∈0,1w \in {0,1} that the structural agent selects per instance. They are switches, not tuned coefficients, and they are activated only when the corresponding layout property holds.
  • The terms are on comparable scales: LolL_{ol} is normalized by object volume, and the IoU-based terms are bounded.
  • All hyperparameters (Adam, learning rate 10−310^{-3}, 300 iterations) are shared across all objects and datasets, with no per-object tuning.

Regarding pipeline complexity: each stage has a single role (initialization, structural refinement, asset synthesis). The stages are decoupled through the proxy, so individual modules can be replaced.

W2. The bounding boxes serve as the control and layout layer, not as the final geometry. Text specifications become explicit on them (counts, positions, hinge axes), and the detailed part meshes are fitted to them. Because each part mesh is generated independently, assembling the meshes without a shared layout constraint can cause interpenetration and collisions during articulation, e.g., a drawer that cannot slide out. The box-overlap penalty is a cheap, differentiable way to prevent this.

Q1. The weights are selected automatically. For each instance, the structural agent AstruA_{stru} analyzes the input image and answers three binary questions: whether the movable parts are left-right symmetric, whether they are centered (and along which axes), and whether they fully cover the front face. Each positive answer sets the corresponding weight to 1, and each negative answer sets it to 0. We will add this description to Sec. 3.3.1:

"The weights wsym,walign,wcov∈0,1w_{sym}, w_{align}, w_{cov} \in {0,1} are not set manually. For each instance, AstruA_{stru} determines from the input image whether the movable parts are mirror-symmetric about the vertical center plane, whether they are centered along each axis, and whether they jointly cover the entire front face; each positive answer sets the corresponding weight to 1. The mandatory terms always use unit weight, and all hyperparameters are fixed across datasets."

Q2. Thank you for the careful observation. Handle placement is determined at the part-generation stage (Sec. 3.4). Part assets are generated by the image generator and SAM3D, and they are reused only through identity or reflection operations.

Because the joint axis of every part is explicit in our proxy, this can be avoided by enforcing that handles lie on the edge opposite the hinge axis, or by adding rotation to the reuse operations. Owing to the modular design, stronger part-generation models (better than SAM3D) can also be substituted directly as they become available.

I thank the authors for the response. However I do not think the response fully addressed my concerns.

Regarding W1, I think what the authors are suggesting is in fact "tuning is not used in the current optimization", rather than "tuning is not required in the current optimization". To support the latter argument, experiments are needed to confirm "the optimization results are not sensitive to weights".

As for W2, my concern remains on penalizing overlapping bounding boxes during layout planning. While I agree that such approach is "cheap" and "differentiable", I do not think it has the potential to support high-quality generation for more complex objects.

I will maintain my current rating.

Reviewer_YLjhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CRAFT, a training-free method for generating 3D articulated objects conditioned on both images and articulation requirements specified in text prompts, with an emphasis on user-specified articulation control. The core method leverages LLMs, including GPT 5.6-Luna and Gemini-2.5-flash, to decompose the desired object based on the input images, ground its functional parts, and specify articulation configurations between them. It then uses SAM3D to reconstruct the geometry of each part and assemble the final output. The parameterization defined in this work supports editing articulation configurations by passing the updated text descriptions back to the decomposer agent. Empirical results show a clear advantage of CRAFT over other articulated object generation methods on Objaverse-Household and competitive performance against trained methods such as SINGAPO on PartNet-Mobility.

Critical strengths
  1. This method takes articulation requirements as input, which is important for articulated object generation because it allows us to control the most critical feature that distinguishes articulated object generation from ordinary 3D generation. The examples shown in Figure 4 also demonstrate the method’s effectiveness in controlling articulation features.
  2. The authors use a structural optimization procedure to refine the preliminary bounding boxes for each part. This procedure appears novel and effective in generating plausible geometry, with ablation results supporting improvements in geometric accuracy.
Critical weaknesses
  1. The experiments do not include quantitative comparisons with other baseline methods on articulation control, using metrics such as diversity and instruction following. The effectiveness of the claimed articulation control is demonstrated only through a limited number of examples, which limits the assessment of the method’s effectiveness and generalizability.
  2. The quantitative evaluation on PartNet-Mobility covers only 7 of the 46 categories in the original dataset. And the existing experimental results do not sufficiently demonstrate the method’s ability to generalize across diverse object geometries.
Questions for authors
  1. Could the authors quantitatively compare the method with other approaches on articulation control tasks, using metrics such as diversity and instruction following?
  2. Could the authors perform articulation control tasks in a wider variety of scenarios and explore the boundaries of the method’s ability to control articulation features? For example, you can investigate for which object categories and under what specific conditions articulation control may fail during generation, or verify that the method can satisfy a diverse range of requirements.
  3. Could the authors test the method’s generalizability, for example by generating a wider variety of articulated objects on PartNet-Mobility or other datasets, or by showing reconstructions of articulated objects from real-world images?
2 comments

We thank the reviewer for the positive assessment and the helpful suggestions.

W1. Our goal is not stochastic diversity. Once a user specifies "three sliding drawers", an output with a different structure is an error, not diversity. Image-only input admits several plausible articulations, and we use text precisely to remove this ambiguity. The diversity in Fig. 1 refers to controllable diversity: different instructions yield different valid configurations from the same image (Fig. 4).

For instruction following, please see our response to Reviewer 1 (W1). We adopt the standard metrics of prior work. A quantitative instruction-following evaluation lacks a benchmark with paired instructions and ground-truth assets, and VLM judges are unreliable for kinematic properties.

W2. We follow established protocols: SINGAPO's PartNet-Mobility split (77 objects, 7 categories) and UnfoldArt's Objaverse-Household split (53 objects, 12 categories). The 7-category split is the in-domain set of the supervised baseline SINGAPO. We test generalization on Objaverse-Household, which has more complex geometry and unseen categories and motions, and on real IKEA product images.

Table 1 supports generalization. From PartNet-Mobility to Objaverse-Household:

  • SINGAPO's rest-state gIoU error rises from 0.570 to 1.213.
  • PhysX-Anything's joint accuracy drops from 34.7% to 19.9%.
  • CRAFT remains robust (gIoU error 0.592 → 0.762; joint accuracy 87.2% → 71.0%) and is best on most metrics.

In addition, CRAFT is training-free, and its priors are defined per part type (Table 3) rather than per object category.

Q1. Thank you for the question. A fair quantitative comparison on articulation control is difficult at present. Most image-based baselines (SINGAPO, PhysX-Anything, FreeArt3D, UnfoldArt) cannot take articulation instructions, and no benchmark pairs images, instructions and ground-truth assets. We will pursue this in future work.

Q2. CRAFT supports the following:

  • rigid parts with fixed, revolute, prismatic and continuous joints (Sec. 3.1);
  • tree-structured kinematics, e.g., handles inherit the motion of their door (Appendix A.1);
  • per-part control over count, placement, joint type, hinge side and opening direction (Fig. 4).

Its current boundaries are:

  • multi-DoF or coupled mechanisms (e.g., screw joints, linkages);
  • deformable parts;
  • optimized joint limits, since motion ranges are predefined (Sec. 5).

Failures mainly come from three sources:

  • The underlying generator (SAM3D). Part details and affordances can be inaccurate, e.g., offset handle positions or parts lacking graspable features. This can hinder manipulation in downstream physics simulation.
  • Strongly non-box-like geometry, which the box proxy approximates only coarsely.
  • Ambiguous instructions, which are resolved by default priors. We will add a discussion of these failure cases in the revision.

Q3. Fig. 3 (columns 5–6) shows results on real product photographs from the IKEA website, where CRAFT produces coherent structures with plausible articulation. Objaverse-Household further contains unseen categories and motions, where CRAFT achieves the best results on most metrics (see W2).

Thank you for your reply. I'd like to keep my rating.

Reviewer_9omSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CRAFT, a framework for text-guided articulated 3D object generation from a reference image. Rather than training a new 3D generative model, CRAFT adopts a modular, model-driven pipeline that integrates existing LLM/VLM agents, pretrained generative models, and explicit geometric optimization. At its core, an editable articulated 3D proxy decouples structural and kinematic control from detailed geometry generation, enabling a multi-stage process involving structural initialization, adaptive optimization, and part-level mesh reconstruction.

Critical strengths
  1. Jointly conditioning on images and textual articulation specifications enables more flexible and user-controlled object generation.
  2. The editable articulated proxy provides a practical interface for controlling part structures, joint configurations, and geometry generation.
Critical weaknesses
  1. The main quantitative evaluation relies on image-derived descriptions rather than independently specified textual instructions. Text-conditioned controllability is demonstrated only through qualitative examples, leaving the paper's central claim insufficiently substantiated.
  2. The individual contributions of geometric constraints and SAM3D-based guidance are not clearly established. In particular, the benefits of adaptive constraint selection over uniformly applying constraints remain inadequately demonstrated.
  3. The framework primarily integrates existing reconstruction models, hand-crafted geometric constraints, and part-level composition. Without comparisons against simpler image-to-3D pipelines augmented with LLM-based articulation editing, it is difficult to assess the necessity and technical significance of the proposed design.
Questions for authors

More quantitative evaluation on textual instructions input;

2 comments

We thank the reviewer for the constructive comments.

W1. We follow the standard protocol of prior articulated-object generation work (SINGAPO, FreeArt3D). It covers geometry (gIoU, centroid and Chamfer distance), articulation (axis error, pivot error, joint accuracy) and plausibility (AOR).

Quantitative evaluation of independently specified instructions is non-trivial. No benchmark pairs images, independent articulation instructions and ground-truth assets. VLM-based judging is also unreliable for kinematics. Hinge side, axis direction and motion range are hard to infer from static renders, and interior parts such as drawer boxes are occluded. We therefore demonstrate text control qualitatively (Fig. 4), and we will discuss this evaluation gap in the revision.

W2. Applying all constraints uniformly (w=1w=1 in Eq. 8) is possible but harmful. Each optional term encodes a layout assumption that holds only for some objects:

  • LsymL_{sym} assumes mirrored left and right parts.
  • LalignL_{align} assumes centered parts.
  • LcovL_{cov} assumes the movable parts cover the whole front face.

When an assumption is violated, the term pulls the proxy toward a wrong layout. For example, LsymL_{sym} would distort "an upper-left hinged door and three sliding drawers" (Fig. 4), and LcovL_{cov} would distort objects whose lids lie on the top face (the rice cooker and trash bin in Fig. 3). The structural agent therefore activates only the terms that apply. Table 2 supports this design. Compared with optimization without the adaptive mechanism, adaptive selection reduces:

  • gIoU error from 0.712 to 0.592;
  • centroid distance from 0.099 to 0.089;
  • Chamfer distance from 0.037 to 0.025.

SAM3D-based guidance plays a different role. It provides the object extent used for depth initialization (Alg. 1) and the multi-view targets for LSAML_{SAM}. LSAML_{SAM} is the only term that ties the proxy to visual evidence; without it, the geometric priors alone would drive the optimization.

W3. An image-to-3D + LLM pipeline can only assign joints to the geometry the image shows. It cannot do two things CRAFT needs:

  1. Change the part structure. This is our core use case. In Fig. 4, the same nightstand becomes "two drawers and a bottom drop-down door" or "one top drawer and two hinged doors", which requires different part geometry.
  2. Produce hidden functional geometry. Image-to-3D models output a fused surface. A segmented "drawer" has no box behind its front panel, and the body has no cavity for it to slide into. CRAFT generates each movable part as a complete asset with a category-informed depth (e.g., 0.9D0.9D for drawers, Sec. 3.2.2) and builds the base from the remaining region (Sec. 3.4).

Close variants of this pipeline are already among our baselines. PhysX-Anything combines TRELLIS geometry with a fine-tuned VLM for articulation, and Articulate-Anything uses a VLM actor–critic loop. CRAFT outperforms both in joint accuracy: 71.0% vs. 19.9% and 57.5% on Objaverse-Household, and 87.2% vs. 34.7% and 72.3% on PartNet-Mobility.

Our contribution is not the individual foundation models, which are interchangeable. It is the editable articulated proxy, which makes textual specifications enforceable before asset synthesis, together with the adaptive structural optimization.

Q1. Please see W1. We adopt the full set of metrics used by existing articulated-object generation methods. Reliable quantitative evaluation of free-form instructions requires a dedicated benchmark with paired instructions and ground-truth assets, which does not exist yet. We consider building one an important direction for future work.

Thank you for the clarification. While the response provides reasonable motivations for the proposed design, my main concerns remain:

  1. Although existing benchmarks lack paired instructions and ground-truth assets, key aspects such as part count, joint type, and axis direction can be evaluated directly from the generated articulation parameters. The central controllability claim still lacks quantitative validation.
  2. Table 2 shows overall improvements but does not isolate the contributions of individual constraints, adaptive selection, or SAM3D-based guidance. Their necessity remains insufficiently demonstrated.
  3. The limitations of direct image-to-3D reconstruction are reasonable, but existing baselines do not directly evaluate the proposed text-controlled structural editing task. Improvements in joint accuracy alone cannot establish the advantages of the more complex pipeline over simpler LLM-based part generation and assembly approaches. Overall, the rebuttal mainly provides conceptual justifications without additional evidence addressing these concerns. I maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.