acceptodds
Withdrawn from ICLR 2027

RelArt: Relation-Aware Articulated Object Synthesis from Executable Structural Priors

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing methods for articulated object generation often predict joint structure and part geometry together from limited observations. The observations may not determine hidden joint axes or the space needed for parts to move, and errors in the predicted structure can cause disconnected parts or collisions. We therefore use an executable structural prior to specify the articulation structure instead of inferring it from the observation alone. We study geometry generation conditioned on this prior, which specifies parts, joints, interfaces, and motion constraints. Our framework, RelArt, includes an agent-based module for constructing these priors and an autoregressive model for generating part geometry. The construction module derives relations between parts from template code and a small set of valid parameter instances. It executes each proposed instance, checks the relations, and repairs parameters that affect failed checks. The verified relations also form the graph supplied to the geometry model. When generating a part, the decoder retrieves its relations to previously generated neighbors and pairs each neighbor's relations with that neighbor's generated geometry. This gives the decoder both the required connection and the geometry to which the new part must fit. With the same image and structural prior, RelArt improves F-score from 0.733 to 0.788 and motion success from 0.781 to 0.869 over the strongest existing generator in our comparison. In a separate construction experiment with equal proposal and executor-call budgets, repairing parameters that affect failed checks produces more valid articulated instances and covers more templates than resampling all parameters.

Primary area:
generative models
Keywords:
Articulated object generation, executable structural priors, autoregressive 3D generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_k6LsOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a new method for articulated object generation. The main idea is to leverage agents to derive the articulated part relations. And then condition the generation on the derived relations. Small scale experiments show the benefit of the proposed approach. However, the overall process does not surprise me given that the agentic 3D reconstruction pipeline has already been widely adopted, and the main competitor/baselines listed in the experiments are at least one year old.

Critical strengths

The paper includes a lot of details in the main document and appendix, though many of them are a bit hard to fully digest. The experiments are clearly organized and described, making it easy to understand. The ablation in the appendix did show the benefit of each proposed component under the framework of current method.

Critical weaknesses
  1. Lack of novelty. It is not novel to use agents for 3d or articulated object generation. Though the overall framework in this paper looks complex, the core idea is not too different from existing works such as (https://arxiv.org/pdf/2604.10992 and https://arxiv.org/pdf/2410.13882), which are missing in related works and the baselines.

  2. The experiments only include old methods without agents. I think comparing against other agentic approaches or simple GPT-style agents is important to prove the advantages of the proposed method in practice and distinguish with more SOTA methods.

  3. The experiments are done on partNet only, which is synthetic and very small scale. There are other datasets like ACD, GAPartNet and AKB-48 that ca be used for verifying the effectiveness of the method either in larger scale or in zero-shot fashion.

Questions for authors

See the weaknesses part for my major concerns.

1 comment

W1. Novelty relative to ArtiCAD and Articulate-Anything. We appreciate these comparisons and agree that using an agent is not itself a contribution. Our contribution is how the same typed part relations are used in repair and geometry synthesis: we trace failed checks to editable template parameters, then pair the current part's relations with the actual geometry of previously generated neighbors. ArtiCAD instead plans connectors before generating CAD parts and uses cross-stage rollback for design and code errors. Articulate-Anything's retrieval pipeline places existing meshes and iteratively refines their articulation. The main distinction is how the relations guide part-by-part generation, rather than the presence of an agent or a feedback loop.

In the shared-prior comparison, all generators receive the same image and executable structure: part roles, hierarchy, joints, interfaces, and the fixed relation graph. Each keeps its generation backbone; only the input representation differs. For example, NAP receives graph nodes and edge attributes, and MeshArt receives the hierarchy and joint edges with projected relation attributes. No method receives the target mesh or extra annotations at test time.

Generator given the same image and structural priorF-scoreMotion success
MeshArt0.7330.781
RelArt0.7880.869

These scores are unchanged from the original submission. The comparison with global fusion at comparable capacity holds the graph and training setup fixed and uses neither induced relations nor prior repair. Step-wise retrieval improves F-score from 0.764 to 0.788, interface distance from 0.025 to 0.019, and motion success from 0.816 to 0.869. See Sec. 5.2, L354–374 (Table 2, originally Table 1); App. A.7.1, L992–1011 (Table 9); A.7.2, L1198–1212 (baseline inputs); Sec. 5.4, L472–473 (Fig. 4).

W2. Baseline recency and the requested agent comparisons. We agree that dedicated agent baselines strengthen the comparison. The original submission already evaluated URDF-Anything+, posted in March 2026, in Tables 3 and 4 (pp. 19–20), but did not include these agent comparisons. The revision adds recent agent systems and simple LLM constructors. The latter share templates, examples, adapters, the LLM, and budgets, with 200 proposals per method. Coverage uses 104 template groups. The feedback baseline can edit any valid parameter but does not use induced predicates or dependency scopes.

ConstructorValid yieldTemplate coverage
Direct LLM0.490.50
LLM with execution feedback0.640.66
RelArt0.820.84

Table 27 uses method-specific inputs. Articraft and ArtiCAD receive an image and object description; RelArt receives an image and a verified prior. Articulate-Anything is a retrieval-and-articulation reference, not a geometry generator in Table 2. Both rates use all attempts in the corresponding row.

SystemAttemptsExport rateMotion success
Articulate-Anything17879.2%50.6%
Articraft16482.9%55.5%
ArtiCAD17173.1%40.9%
RelArt with a verified prior18391.8%75.4%

On the same 120 web photos, RelArt instead estimates its prior from each photo. With the same reference-motion check, motion success is 54.2% for Articraft and 80.8% for RelArt. Table 5 compares the complete pipelines; Table 2 isolates geometry generation by holding the image and prior fixed. New results: Sec. 5.3, L425–446 (Table 4); App. A.8.5, L1614–1630 (Table 27); Sec. 5.4, L486–503 (Table 5).

W3. Dataset scale and transfer. We agree that the suggested datasets would be useful for testing transfer. Our benchmark uses the kinematically annotated PartNet-Mobility dataset, with 391 objects in the main test split. For context, the full releases contain 2,346 objects in PartNet-Mobility, 354 in ACD, 1,166 in GAPartNet, and 2,037 in AKB-48. These are release sizes, not test counts. GAPartNet reuses PartNet-Mobility assets, so a transfer test would need to exclude training-set objects.

In Table 19 we exclude each test category and its templates during training but allow the same kinds of parts, joints, and relations to occur in other categories. With the same prior at test time and neither relation induction nor prior repair, retrieval improves F-score from 0.695 to 0.755 and motion success from 0.689 to 0.821 over the base model. It also reduces interface distance from 0.035 to 0.026 relative to flat relations. This tests new combinations within PartNet-Mobility. Neither this test nor the web-photo study establishes transfer to the proposed datasets. See App. A.7.7, L1364–1387 (Table 19, originally Table 15); A.11, L1751–1757.

Reviewer_DCLrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RelArt tackles the challenge of generating articulated objects from a single image. It splits the whole task into two separate parts: building articulated structural priors and generating part‑level geometry. Its relation‑induction module uses template code and a small set of valid examples together with a large model to find constraints between parts. It checks results by running programs, and fixes violations inside limited parameter ranges. This way produces more valid articulated objects than resampling all parameters. Its relation‑aware autoregressive decoder creates parts one by one following topological order. It uses both prior constraints and geometry of already‑built neighbor parts, so new parts better fit interface and motion rules. In controlled tests where all baselines share the same input images and structural priors, RelArt achieves higher F‑score and motion success rate, and reduces interface error and part collision. Still, this method has clear limits. It heavily depends on hand‑written executable template libraries and joint adapters. It cannot directly output articulated objects end‑to‑end from real photos. Parameter repair only changes numeric values and cannot fix topology flaws inside templates. Motion safety is only checked on discrete sampled poses, not the full continuous motion range. Though deterministic checks follow large‑model outputs, relation suggestions are still affected by large‑model quality. Main experiments use synthetic rendered images, without full tests on noisy real‑world photos. Overall, its task‑splitting idea and autoregressive generation with geometry history offer useful references. Yet real‑world application still needs solutions for upstream template preparation.

Critical strengths

This work separates structural‑prior construction from part geometry generation to avoid the ambiguity of inferring hidden joint information only from single‑view images. Critically, the authors run controlled comparisons where all baseline methods receive identical input images and shared executable structural priors. This setup isolates geometry‑generation performance and makes comparison fair. Such careful experimental design strongly increases trust in the reported improvements and shapes my positive overall assessment.

Critical weaknesses
  1. The whole pipeline requires pre built templates and joint adapters. It cannot take a raw real world photo and directly produce an articulated object end to end. Template creation work is excluded from runtime and evaluation. This limits real world applicability heavily. Even good geometry generation performance cannot compensate for the need for large manual upstream work, and this weighs negatively for practical value of the method.
  2. All main evaluations use clean synthetic rendered images instead of noisy real photographs. Besides, the initial relation proposals rely on an LLM. Although later deterministic execution checks filter bad outputs, wrong relations from the LLM will hurt final results. Perturbation tests already show that errors in prior relations degrade motion related metrics. Since downstream geometry generation cannot fix broken upstream priors, the overall system robustness is questionable for practical use cases.
Questions for authors
  1. In your current setup, hand‑written templates and joint adapters are required inputs. Could you quantify how much manual engineering time is needed to make new templates and joint adapters for a new object category? This will help us better judge the real‑world cost of your pipeline.

  2. Your perturbation ablation shows that wrong relation attributes or missing relation edges hurt final performance. When the LLM outputs invalid relation proposals that pass the executor check but do not match real‑world object rules, what practical failure rates do you observe on held‑out templates?

  3. All main experiments use clean synthetic rendered images. If you feed real noisy photos as input, can you provide quantitative results for the full pipeline? If not, what main technical blocks stop you from running these real‑image evaluations?

1 comment

W1 / Q1. How much human work is needed to write a template or joint adapter? A VLM-based coding agent can draft parameterized templates and joint adapters instead of having an expert write each one by hand. In the pilot, the agent received assembly specifications and the geometry API and wrote both the constructors and joint records. We drew 24 parameter sets per template and checked mounting alignment and penetration at 17 motion states per joint. All 48 instances passed. This demonstrates code authoring from a supplied specification; the timing studies below answer how much human work it saves.

The timing studies distinguish three tasks: reuse an existing adapter, write an adapter for an already supported joint type, and add support for a new joint type. With identical specifications and tests, we measure mean active human time for writing code or prompts, debugging, and verification. This excludes specification and test preparation and is separate from per-object runtime.

Authoring taskTasks per conditionManual (min)Agent-assisted (min)
Reuse an adapter1415.55.8
Write an adapter for a supported joint type963.024.0
Implement a new joint type6155.068.4

The last row covers three helical-joint tasks and three rack-and-pinion tasks. Both modes passed all six, including motion-transform, export, and regression checks. Assistance reduced active human time by 55.9% in these tasks. Revised PDF: App. A.8.1, L1501–1519 (Table 21); A.9, L1646–1654 (authoring pilot); A.10, L1732–1749 (Table 31).

W2 / Q2. How often do incorrect relations pass the executor and cause asset failures? We audited 389 assets that passed execution checks, all from held-out templates with joint adapters. An accepted relation is not necessarily correct: the executor checks whether its predicate holds, whereas the audit asks whether it is the right relation for the object.

Audit outcomeCountRate
Incorrect relation among audited assets5113.1%
Interface or motion failure among those with an incorrect relation3670.6%
Pass after correction among the 36 failures2877.8%

The 36 failures are 9.3% of all audited assets. To see whether correcting the relation changes the output, we replaced the wrong relation with an independently verified one, or removed it if no correct relation existed. We reran the same decoder, holding the image, template parameters, weights, random state, and reference fixed. The correction is supplied by the audit; the recovery rate does not measure automatic repair. See App. A.8.3, L1578–1590 (new Table 25).

Q3. Can the full pipeline work from real photos? Both systems use the same 120 web photos, one per object after deduplication. RelArt does not receive a human-verified prior: GPT-6 estimates a template and part and joint descriptions from each photo. We check the inferred prior before generating geometry. Both outputs must include the required moving parts and complete the same reference motion without prohibited collisions at sampled states. The denominator is all 120 photos per system, including missing exports. The comparison fixes the photos and scoring, not the systems' prompts or search budgets.

SystemMotion passesMotion success
Articraft6554.2%
RelArt with automatic priors9780.8%

The 32 additional passes give a 26.7-point gap based on the counts. The result depends on both prior estimation and synthesis. A photo whose articulation is not covered by the library remains outside the current estimator's scope.

We also run RelArt with automatic priors on renders of the 391 PartNet-Mobility test objects. It produces 271 evaluable assets (69.3%) and 232 motion passes (59.3%), both measured over all 391 images. An output is evaluable when its part geometry and joint transforms can be scored. Dataset joint annotations define the reference motion for scoring and are not given to the generator.

A separate test asks how photographs affect the generator when the prior is already known. We evaluate 120 objects across 10 viewpoints using paired photographs and renders. Each pair shares the viewpoint and articulation state, and the prior and model weights stay fixed. Motion success goes from 73.5% on renders to 66.7% on photos for the base generator and from 86.2% to 79.3% for RelArt. RelArt remains 12.6 points ahead on photos, although the two models drop by similar amounts. This test isolates the change in image input; it does not measure prior estimation from a photo.

Revised PDF: Sec. 5.4, L475–481 (main-text summary); L486–503 (Table 5, web photos); L505–515 (Table 6, image-only renders); L518–533 (Table 7, fixed-prior photo–render comparison).

Reviewer_jgKvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

RelArt separates structural inference from geometry generation for articulated objects. It has two parts:

  1. An agent-based module expands reusable parametric templates into verified structural priors. An LLM proposes inter-part relations, an executor checks them, and failed checks are repaired only within parameter scopes traced from the template's AST.
  2. An autoregressive SDF decoder conditions each part on its relations to already-generated parts and on their generated geometry.

With the same image and prior, RelArt improves F-score from 0.733 to 0.788 and motion success from 0.781 to 0.869 over the strongest baseline. Scoped repair also beats full resampling, 0.82 vs. 0.74 validity.

Critical strengths
  1. Verifiable, traceable LLM-based structure construction. The LLM only proposes relations; an executor decides, and repair scopes are traced from the AST. This directly addresses agents inferring wrong structure. Removing the executable checks drops validity from 0.82 to 0.57 (Table 12). This is the paper's most transferable idea.

  2. Relation- and history-conditioned AR generation, supported by controlled ablations. Step-wise retrieval beats a capacity-matched global-fusion variant (F 0.788 vs. 0.764, Table 5). Removing the generation history drops F to 0.759. So the gain comes from the mechanism, not from extra capacity.

Critical weaknesses
  1. Heavy reliance on hand-authored priors, with their impact left uncharacterized. What the method can cover is limited by hand-written templates, joint adapters and the supported relation set. The paper does not evaluate unseen joint types or out-of-library inputs, and its reported cost leaves out the effort of authoring templates and adapters. It is therefore unclear where priors for genuinely new objects would come from, and how much the results depend on getting those priors right.

  2. No methodological account of preconditions or transferability. The method's preconditions (static traceability, a valid joint adapter, relation coverage) are mentioned only in scattered places and are never quantified. The decoder design is validated only on the authors' own backbone. There is also no comparison with agent-only structure inference such as Articraft, and no analysis of whether the verification module would help such pipelines. Because the system is engineering-heavy, it remains doubtful that its design choices would give consistent gains when applied to other methods.

Questions for authors
  1. How large is the template library, and what does adding a new category or joint type cost? Are out-of-library inputs detected?
  2. How sensitive are results to errors in the structural skeleton, such as part partition or joint axes?
  3. What does the "Without executable checks" variant retain? Can you add a pure-LLM construction baseline?
  4. A breakdown of failures and repair success by type (parsing / geometry / joints / relation).
  5. Evidence for why scoped repair beats full resampling, e.g., an equal-size random-subset control.
3 comments

W1 / Q1. How large is the library, what does it cost to extend, and how are unsupported requests handled? The source library covers 39 object categories. We grouped near-duplicate programs and used 104 template groups for the LLM comparison; 104 is not the full library size. A VLM-based coding agent can draft template code and joint adapters. Given assembly specifications and the geometry API, it wrote two templates whose 48 instances passed interface and sampled-motion checks. With specifications and tests supplied, assistance cuts mean human time from 15.5 to 5.8 min when reusing an adapter, from 63.0 to 24.0 min when writing one for a supported joint type, and from 155.0 to 68.4 min across six tasks implementing new joint types. Timing excludes specification and test preparation.

When the requested parts and joints are specified, we check for a compatible template and verified adapter. Of 26 supported requests, 21 yield articulated assets, 4 yield geometry only, and 1 is rejected. Of 47 unsupported requests, 8 yield geometry without articulation and 39 are rejected. This study does not test detection from a photograph alone.

Revised PDF: App. A.8.1, L1495–1534 (Tables 20–22); A.9, L1646–1654 (authoring pilot); A.10, L1732–1749 (Table 31).

W2. What do the checks require, and do the decoder and verifier transfer? Relation checks need supported rules and identifiable parts; motion checks also need a joint adapter. Repair needs parameters that can be traced to the failure. We find them for 90.6% of 1,059 parameter, geometry, joint, or relation failures; this excludes syntax failures and unclassified cases (Table 24). With five examples, 80% of audited templates have at least one correct relation that passes the checks. On ArtFormer, step-wise retrieval improves F-score from 0.720 to 0.744 and motion success from 0.780 to 0.819 on 240 objects, with the inputs, geometry history, and decoder capacity held fixed. On the same 120 photos, RelArt reaches 80.8% motion success and Articraft reaches 54.2%.

Of 136 Articraft exports, 115 could be converted and scored independently. Before screening, 76 were usable. Screening retains 80 assets, of which 71 are usable. It raises the usable share among kept assets from 66.1% to 88.8%. It also rejects five usable assets, so over all 164 attempts, usable yield falls from 46.3% to 43.3%. No assets are modified.

Revised PDF: App. A.7.3, L1243–1272 (Table 14 audit); A.8.6, L1632–1643 (Table 28); A.10, L1714–1730 (Table 30); Sec. 5.4, L486–503 (Table 5).

Q2. Sensitivity to prior errors. We perturb the priors of 180 objects with the images and decoder weights fixed, regenerate geometry, and score it against the original reference motion. Axis rotations of 5°, 15°, and 30° lower motion success by 1.2, 5.6, and 14.3 percentage points. Pivot shifts of 2%, 6%, and 10% of the object diagonal cost 0.9, 3.4, and 7.8 points, respectively. Merging or splitting slots costs 17.5 or 12.1 points. See App. A.8.4, L1593–1611 (new Table 26).

Q3. What remains without checks, and how does an LLM alone perform? The variant retains relation induction, template instantiation, the proposal budget, and independent scoring. It removes construction checks and the repair they would trigger. Predicate satisfaction is reported as a diagnostic, not used as a filter. Validity is 0.57 without checks or with filtering alone, 0.74 with full resampling, and 0.82 with scoped repair. Under equal template access and budgets, direct LLM, LLM with execution feedback, and RelArt reach yields of 0.49, 0.64, and 0.82 and coverages of 0.50, 0.66, and 0.84, respectively. See Sec. 5.3, L425–446 (new Table 4); App. A.7.5, L1285–1304 (clarification accompanying Table 16).

Q4. Which failures are recovered? Counting each candidate at its first failed stage, AST-guided repair recovers 348 parameter, 171 geometry, 163 joint, and 146 relation failures: 71.9% of all 1,151 failures. The source-rewrite fallback recovers 63 of 69 parsing failures under the same final check and is counted separately. See App. A.8.2, L1556–1575 (new Table 24).

Q5. Do the gains come from choosing the right parameters? We use the same 181 failures from held-out programs on which all three policies can run and cap each policy at 9,600 executor calls. Both local policies use the same update rule and select the same number of parameters; random selection is stratified by parameter type. The table reports actual calls, not the allowed maximum.

PolicyRecoveryExecutor calls
Full resampling57.5%9,412
Random choice, same number of parameters68.0%9,468
AST-guided choice81.2%9,336

AST-guided selection recovers 24 more cases than random selection with fewer calls. See App. A.8.2, L1536–1554 (new Table 23) and A.9, L1691–1695 (new Fig. 13, examples with injected mounting errors).

Reviewer_jgKvOpenReview ↗

Suggestions for Revising the Original Text

Is it possible to highlight the revisions in the revised manuscript using a distinct color or other markers? It is very difficult to identify the changes if they are only mentioned in the response.

Sure. We have updated the revisions and highlighted the parts modified in the rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.