acceptodds
Under review as a conference paper at ICLR 2027

AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents

20 views

Abstract

The task of 3D assembly requires translating an understanding of parts and their relationships into precise spatial arrangements. Can pretrained general-purpose agents assemble objects through visual interaction without additional assembly-specific fine-tuning? To investigate this question, we introduce AssemblyWorld, an interactive 3D environment in which agents inspect rendered views and manipulate supplied rigid parts, guided by images or assembly manuals when available. Agents perceive part geometry through 2D views rather than direct access to mesh vertices or faces, while their resulting assemblies are evaluated geometrically. Building on this environment, we construct AssemblyWorldBench, comprising 100 assembly tasks across 80 objects spanning furniture, industrial assembly, and fracture reassembly. Evaluating eight agent systems reveals substantial differences in their capabilities. The strongest system achieves 80.9% part accuracy but 59.4% complete-assembly success. The evaluated open-source systems lag substantially behind their stronger closed-source peers in both execution reliability and assembly accuracy. Analyses of visual references, interaction trajectories, and failures show how agents revise assemblies while leaving residual positioning errors. AssemblyWorld provides a common setting for both assessing the capabilities of interactive assembly agents and characterizing the gap between approximate structure recovery and precise reconstruction.

Primary area:
datasets and benchmarks
Keywords:
3D Assembly, 3D Understanding, Multimodal Agents, Benchmarks

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_U9wrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AssemblyWorld, an interactive environment for assembling a fixed set of rigid 3D parts using pretrained general-purpose agents without additional assembly-specific fine-tuning. Agents inspect rendered views and structured scene records, select viewpoints, and manipulate individual parts or groups through MCP tools. Part geometry and relative scale remain fixed; mesh vertices, faces, target poses, and evaluator annotations are not exposed through the interface. Numeric pose and bounding-extent queries are available, so the observation setting combines images with structured metadata.

AssemblyWorldBench contains 100 tasks over 80 distinct objects from PartNet, IKEA-Manual, AssemblyBench, and Fantastic Breaks. PartNet objects have paired image/no-reference conditions; the other sources use their specified manual or no-reference setting. Eight model–harness systems are evaluated with common initial configurations, tools, task text, and a 60-minute benchmark execution budget. Geometric evaluation uses one global rigid alignment, matching of equivalent parts, shape Chamfer distance, part accuracy (PA), and all-parts shape success (SR). The best observed system, GPT-6 Astra, achieves 80.9% PA and 59.4% SR under equal-source weighting.

Larger source-level evaluations, paired reference comparisons, trajectory and failure analyses, task perturbations, and a matched bidirectional agent–GARF experiment provide additional evidence. The contribution is a common evaluation environment and an empirical characterization of interactive assembly, with demonstrated results limited to free-space geometric reconstruction.

Critical strengths
  1. A useful, clearly delimited benchmark for interactive spatial reasoning across assembly domains. The same interface requires agents to connect visual inspection, numeric actions, and verification of the resulting geometry across furniture, industrial components, and real fracture fragments. Fixed part geometry and scale make successful reconstruction assessable, while global alignment and equivalent-part matching accommodate irrelevant overall pose and interchangeable components without independently repositioning parts during scoring. Paired PartNet reference conditions and shared initial states make within-benchmark comparisons interpretable. The setting also complements specialized multimodal pose prediction such as the already-cited AssemLM, which uses a dedicated point-cloud encoder. This combination of task definition, interaction, and cross-domain evaluation is the main reason I support acceptance.

  2. The empirical analysis provides useful evidence about both successful assembly and remaining failure mechanisms. The paper separates partial part placement from complete reconstruction and examines how reference information, intermediate edits, and residual errors relate to the outcomes. The appendix preserves object pairing in uncertainty estimates and explicitly states that the Astra–Fable difference is not significant after the reported multiple-comparison correction. Particularly useful is the matched four-way experiment: applying GARF after the agent reduces PA from 91.67% to 87.00%, whereas GARF followed by the agent reaches 93.00% on the same 150 two-fragment objects, with improvements and regressions reported per object. The negative composition result gives concrete guidance for future refinement methods. Together with the larger single-agent evaluations in Table A-1, these analyses add research value beyond the common benchmark's aggregate scores.

Critical weaknesses
  1. The comparisons with specialized methods do not sufficiently establish performance advantages under a common evaluation protocol. Section 4.3 and Tables 2–5 compare Astra with results obtained under the specialized methods' reported settings. Appendix A.3 explicitly states that the Fantastic Breaks alignment and sampled inputs do not constitute a matched reproduction of the published runs, and that the IKEA alternative success metric, SRaSR_a, uses each method's own normalization. Consequently, the same numerical distance threshold need not represent the same geometric tolerance. For the furniture and industrial comparisons, it is not sufficiently clear that normalization, point sampling, global registration, equivalent-part assignment, and initialization differences have been reconciled. This is especially consequential for interpreting Table 3's 100% SRaSR_a and the PA/SR advantages emphasized in Section 4.3. These results demonstrate capability, but the claim of superiority to specialized methods needs at least a representative comparison with a shared evaluator or narrower wording. This concern does not invalidate the common eight-system comparison or the internally matched experiment in Table 6.

  2. Visual-input validity remains unresolved for part of the low-performing system comparison. Appendix C.2 documents a DeepSeek run that reports receiving no viewable image despite image payloads in the tool records; whether those images actually reached the model through its provider remains unresolved. Qwen and DeepSeek also have substantial execution failures and timeouts. Counting such failures is appropriate for an end-to-end system benchmark, and the paper correctly identifies the evaluated units as model–harness combinations. Nevertheless, successful image delivery is a prerequisite for interpreting the results as evidence about visual assembly capability. Identical tool schemas do not establish that prerequisite across provider adapters. An end-to-end check of image consumption and basic observation/action functionality is needed to determine whether the low scores reflect assembly limitations, integration faults, or both. This limits the interpretation of the reported gap involving these systems; it is not evidence that image delivery failed in every run.

Questions for authors
  1. Clarification and targeted validation of the specialized-method comparisons. Which rows in Tables 2–4 share Astra's object set, reference condition, normalization, point sampling, global alignment, equivalent-part matching, and success threshold, and which retain a different protocol? In particular, how should Table 3's SRaSR_a values be compared when normalization is method-specific? For at least one representative furniture or industrial baseline with available predictions or an implementation, can you report results using a shared evaluator and clearly specified input/initialization conditions? Whether the reported PA/SR advantage persists would materially affect my assessment of the comparative evidence.

  2. Clarification and targeted validation of provider image delivery. Can you establish whether Qwen and DeepSeek receive usable camera images through the evaluated adapters, including the unresolved case in Appendix C.2? Please describe an end-to-end check that demonstrates image consumption and a simple observation–action–observation cycle, and quantify any affected runs identifiable from the records. If a delivery or adapter problem is found, results on a fixed, representative subset after correction would help determine whether the reported system gap persists. The answer would materially affect my confidence in the interpretation of the low-performing systems.

3 comments
AuthorsOpenReview ↗

Response to Reviewer U9wr

W1 / Q1. Evaluation comparability

We clarify that the differences in evaluation implementation concern three aspects: point sampling, global alignment, and scale normalization. For the matched comparisons reported in Tables 2–4, we retain the same PA/SR definitions, success thresholds, and equivalent-part matching principles. These differences arise from evaluating assemblies from different data sources in a common interactive environment. Below, we explain each difference and its effect on the reported results.

Point sampling

Geometry and pre-sampled point clouds are not provided uniformly across sources. We therefore generate evaluation point clouds from meshes using a common procedure and fix the random seed for reproducibility. On five IKEA-Manual objects with fixed final assemblies, we resampled and rescored the results across three seeds. Mean SCD ranged from 3.558×10−43.558 \times 10^{-4} to 3.664×10−43.664 \times 10^{-4}, with a maximum relative change of approximately 3.03.0\\% from the original sampling seed, while PA, SR, and SRᵃ remained unchanged for every object.

Global alignment

Existing methods typically predict poses in a specified target coordinate frame, whereas AssemblyWorld allows agents to assemble objects in a free coordinate frame. We therefore apply a single rigid transformation to the entire assembly. This alignment removes only the arbitrary global coordinate frame of an assembly. It does not independently reposition parts or correct their relative placement errors. In the two-fragment comparison with GARF, both configurations use the same fixed-reference-part anchor alignment.

Normalization

We normalize scale during offline evaluation to apply a consistent scoring convention across different sources. Applying the normalization convention described by AssemLM to our predictions leaves SRᵃ unchanged at 100100\\% across all 102102 IKEA objects. The maximum SCD is 0.007280.00728, well below the shared threshold of 0.020.02. Thus, the normalization difference does not explain the 100100\\% SRᵃ reported in Table 3.

The sampling and normalization checks therefore preserve the corresponding reported success metrics, while global alignment accounts for the arbitrary overall pose permitted by the interactive setting without changing the assembled structure.

W2 / Q2. Image delivery and assembly replays

We conducted five end-to-end image checks each for DeepSeek and Qwen, covering the initial reference image, manual images, and camera screenshots in a cycle of observation, action, and further observation. Both systems passed all checks, and the scene records confirmed a valid part displacement. Each system then replayed five fixed assembly cases. DeepSeek had PA and SR both equal to zero in all five cases and made no part-pose edits, remaining mainly in tool and scene queries. For Qwen, one run terminated normally and four timed out. All five obtained PA=0\mathrm{PA}=0 and SR=0\mathrm{SR}=0. These results show that, in the current configuration, usable images alone do not ensure effective assembly.

Dear Reviewer U9wr,

Thank you again for your careful review and for raising the questions about evaluation comparability and image delivery. In our response, we clarified the evaluation protocols used for comparisons with specialized methods (W1/Q1) and reported end-to-end image-delivery checks and assembly replays for DeepSeek and Qwen (W2/Q2).

We hope these clarifications are helpful. If either point would benefit from further discussion, we would be happy to follow up before the discussion closes.

Best regards, The authors of Paper 35762

Thanks for the additional experiments to clarify my concerns. I would like to maintain my score.

Reviewer_Y8APOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AssemblyWorld, an interactive environment for assembling fixed rigid parts through rendered observations and pose-editing tools, together with a benchmark covering furniture, industrial assembly, and fracture reassembly. The paper evaluates eight general-purpose agent systems under shared conditions and conducts larger-scale comparisons between Astra and specialized assembly methods. Further experiments analyze reference use, interaction behavior, and failure modes, and explore combining Astra with GARF for geometric refinement.

Critical strengths
  • The paper introduces an assembly-specific evaluation protocol across several data sources. Fixing the supplied part geometry and scoring final part configurations provides a useful evaluation setting distinct from judging rendered-image resemblance. Shared objects, initialization, tools, and execution budget facilitate internal comparison of the evaluated systems.

  • The paper provides detailed empirical analyses, including larger-scale comparisons between Astra and specialized assembly methods and examinations of reference effects, interaction trajectories, and execution and geometric failures. The composition experiments with GARF further explore the interaction between general-purpose agents and geometric models. These analyses help interpret the observed performance differences.

Critical weaknesses
  • The shared evaluation contains 20 objects per data source, totaling 80 distinct objects. It remains unclear whether the reported system differences are robust to alternative object selections. The larger evaluations primarily assess Astra and therefore do not directly establish the generalizability of the multi-system findings.

  • Limited validated guidance on improving assembly performance. The paper identifies precision and reliability limitations, but its proposed direction of combining agents with geometric models receives only preliminary validation. The GARF experiments are restricted to two-fragment setting, leaving unclear whether the approach consistently improves assembly performance in other settings, which limits the insights derived from the empirical analyses.

Questions for authors
  • Why do the resource measurements cover only 88 runs for Opus, 78 for DeepSeek, and 34 for Qwen? Are the missing records associated with timeouts or execution failures, and how might their exclusion affect the comparability of the mean resource usage reported in Table 1?

  • Do the tool-call errors, stream disconnections, and image-delivery issues described in Appendix C.2 originate primarily from the models, from the execution harnesses or other issues? To what extent do these issues affect the reported assembly scores and the interpretation of the evaluated systems’ capabilities?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Y8AP

W1. Benchmark size and object selection

Thank you for raising the robustness of object selection. All systems used the same objects, reference inputs, initial configurations, tools, and execution budgets. As detailed in Appendix A.1, we sampled 2020 objects from each of four sources using category quotas and part-count stratification where applicable, and aggregated results with equal weight across sources.

The overall success rates of Astra, Fable 5.1, and Opus 5 are 59.459.4\\%, 50.050.0\\%, and 44.444.4\\%, respectively, whereas all other systems achieve at most 11.911.9\\%. These results show clear capability differences under the same interactive assembly setting. The 100100-task scale also balances coverage and affordability: the 8888 Opus 5 runs with recorded costs alone cost approximately USD 1,0891{,}089, while all 100100 Astra runs had an estimated cost of USD 202202 (Table 1).

We further resampled Astra’s existing per-object results using 1,0001{,}000 random seeds, retaining the category quotas and part-count constraints. Each draw selected 2020 objects per source, preserving the paired PartNet conditions to yield 8080 objects and 100100 tasks. Mean PA was 83.2483.24\\% (\pm 2.42\\%), and mean SR was 63.3563.35\\% (\pm 4.06\\%), compared with 80.980.9\\% and 59.459.4\\% in the original benchmark. These results indicate that Astra’s overall assembly performance is not confined to the original object selection.

W2. Geometric refinement and future work

This experiment explores how general-purpose agents and specialized geometric models can complement each other and informs future method design. Our main contributions are a unified environment for evaluating interactive assembly and an empirical analysis of agent capabilities.

Table 6 shows that composition order matters: on the same 150150 objects, applying GARF after the agent reduces PA from 91.6791.67\\% to 87.0087.00\\%, whereas the reverse composition reaches 93.0093.00\\%. Directly chaining the two methods does not guarantee improvement, motivating refinement models designed for the error distribution of agent-generated assemblies.

Q1. Missing resource measurements

Yes, the missing records are predominantly associated with timeouts. Opus, DeepSeek, and Qwen have 1212, 2222, and 6666 missing token/cost records, respectively; 1212, 1717, and 6565 of these are timeouts (Appendix C.2). When a run is terminated, the CLI cannot return its usage. This affects only tokens and their corresponding costs; PA, SR, runtime, and tool-call counts still cover all 100100 tasks.

We estimated missing input, cached-input, and output tokens from runs of the same system and task condition with available records, using runtime and tool-call count as two separate exposure measures, then applied the corresponding model prices. Per-task means are:

SystemRecorded runsOriginal tokens/run (M)Estimated tokens/run (M)Original cost/run (USD)Estimated cost/run (USD)
Opus88/10088/10016.8916.8917.42–18.0917.42\text{–}18.0912.37712.37712.740–13.24212.740\text{–}13.242
DeepSeek78/10078/1002.592.593.27–3.733.27\text{–}3.730.1620.1620.195–0.2200.195\text{–}0.220
Qwen34/10034/1003.673.673.75–3.903.75\text{–}3.901.6741.6741.827–1.8961.827\text{–}1.896

Under both estimates, including the missing runs increases mean token usage and cost relative to the recorded subset. We will revise Table 1 accordingly.

Q2. Execution issues and image validation

As discussed in Appendix C.2, the issues are mainly execution timeouts and a small number of invalid MCP arguments. Some DeepSeek calls supplied arguments that did not satisfy the interface requirements, preventing tool execution; some runs did not complete assembly within the allotted time. Interpreting interfaces, recovering from feedback, and progressing through actions within the budget require both LLM capability and effective coordination with the harness. Automatically rewriting arguments or extending the time limit alone does not guarantee a solution. We retain these runs and score their final states to evaluate the complete agent system’s end-to-end assembly performance.

We also conducted end-to-end image checks each for DeepSeek and Qwen on five failed tasks. Both systems successfully read the images and completed a cycle of observation, action, and further observation. We then replayed five assembly cases per system. Both systems obtained PA=0\mathrm{PA}=0 and SR=0\mathrm{SR}=0. Thus, in the current configuration, usable images alone do not ensure effective assembly.

Dear Reviewer Y8AP,

Thank you again for your thoughtful review and constructive feedback. We wanted to briefly follow up on our response, where we included additional checks on benchmark robustness (W1), discussed the scope of our geometric refinement experiments (W2), and clarified the missing resource measurements and execution issues, including our image-delivery checks (Q1–Q2).

We hope these results and clarifications are helpful. If any of your concerns remain or if there is anything you would like us to clarify further, we would be happy to respond before the discussion closes.

Best regards, The authors of Paper 35762

Thanks for the authors' responses. My concern has been addressed and I will raise my score.

Reviewer_fhquOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

AssemblyWorld studies whether VLM agents can assemble rigid 3D parts through visual interaction without assembly-specific fine-tuning. It provides a common environment for inspecting rendered views, querying scene information, and editing part poses, with geometric evaluation of the resulting assemblies. The paper compares eight agent systems on 100 tasks spanning furniture, industrial assembly, and fracture reassembly. The results demonstrate substantial assembly capability while revealing a gap between approximate structural correctness and complete geometric success.

Critical strengths
  1. Timely evaluation through a well-designed interactive task. Assembly is closely related to the spatial reasoning abilities of vision-language models. The environment provides a reasonable common interface for evaluation of the task.

  2. Extensive experiments with informative analyses. The empirical study combines an eight-system comparison on tasks covering a rich group of objects from PartNet, IKEA, and industrial objects. Results provide useful evidence about both capabilities and remaining limitations.

Critical weaknesses

No critical weaknesses.

Questions for authors

For the IKEA results in Table 3 and the manual-understanding claim at lines 322–323, it would be great to compare Astra under three conditions: (i) no reference, (ii) a single image of the finished object, and (iii) the full manual, keeping the object's initial configuration. The existing PartNet image-versus-no-reference comparison is useful, but does not isolate the additional benefit of the full manual on IKEA task.

2 comments
AuthorsOpenReview ↗

Response to Reviewer fhqu

Q1. Reference information on IKEA

Thank you for this suggestion. On the same 2020 IKEA objects in the main benchmark, we fixed the initial configurations and execution budget and compared three conditions: no reference, a single manual-cover reference image, and the full manual:

ReferencePASR
None82.8882.88\\%5050\\%
Single cover image90.5090.50\\%7070\\%
Full manual89.2889.28\\%7575\\%

The reference image substantially improves assembly performance over no reference on this set. Relative to the single-image condition, the full manual changes two objects from failure to success and one from success to failure, yielding a net SR increase of 55 percentage points, while mean PA does not increase.

The Astra comparisons on IKEA-Manual and PartNet consistently show that visual references can substantially improve assembly performance; in many cases, a single image of the completed assembly already provides the key reference information needed to complete the task.

Dear Reviewer fhqu,

Thank you again for your encouraging feedback and for suggesting the additional IKEA experiment. Following your suggestion, we compared three conditions in our response: no reference, a single reference image, and the full manual, using the same initial configurations. This comparison helped us better understand what each type of reference contributes.

If you have any follow-up questions about the results, we would be happy to address them before the discussion closes. Thank you again for taking the time to review our work.

Best regards, The authors of Paper 35762

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

We thank the reviewers for their constructive feedback. A central question is whether AssemblyWorldBench supports fair comparisons and distinguishes models’ assembly capabilities. Our additional experiments show that the evaluation choices we tested had little effect: changing point sampling and normalization in the IKEA experiments did not alter the main findings. The performance gap was clear even within Codex CLI: on the same tasks, with the same environment and scoring criteria, Astra achieved 59.4% complete-assembly success, while DeepSeek did not complete any task successfully.

More broadly, our comparisons with domain-specific assembly methods point toward a shift in how we approach 3D assembly. Without any assembly-specific fine-tuning, Astra can assemble furniture, industrial objects, and fractured objects. Specialized methods remain valuable for precise part placement. Together, these findings suggest a division of labor: general-purpose agents handle reasoning across domains, while specialized geometric methods refine part poses. Our experiments also show that the order in which these components are combined matters. We hope AssemblyWorld provides a common platform for studying how to make them work together. Further details are in our point-by-point replies below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.