- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces AssemblyWorld, an interactive environment for assembling a fixed set of rigid 3D parts using pretrained general-purpose agents without additional assembly-specific fine-tuning. Agents inspect rendered views and structured scene records, select viewpoints, and manipulate individual parts or groups through MCP tools. Part geometry and relative scale remain fixed; mesh vertices, faces, target poses, and evaluator annotations are not exposed through the interface. Numeric pose and bounding-extent queries are available, so the observation setting combines images with structured metadata.
AssemblyWorldBench contains 100 tasks over 80 distinct objects from PartNet, IKEA-Manual, AssemblyBench, and Fantastic Breaks. PartNet objects have paired image/no-reference conditions; the other sources use their specified manual or no-reference setting. Eight model–harness systems are evaluated with common initial configurations, tools, task text, and a 60-minute benchmark execution budget. Geometric evaluation uses one global rigid alignment, matching of equivalent parts, shape Chamfer distance, part accuracy (PA), and all-parts shape success (SR). The best observed system, GPT-6 Astra, achieves 80.9% PA and 59.4% SR under equal-source weighting.
Larger source-level evaluations, paired reference comparisons, trajectory and failure analyses, task perturbations, and a matched bidirectional agent–GARF experiment provide additional evidence. The contribution is a common evaluation environment and an empirical characterization of interactive assembly, with demonstrated results limited to free-space geometric reconstruction.
Critical strengths
-
A useful, clearly delimited benchmark for interactive spatial reasoning across assembly domains. The same interface requires agents to connect visual inspection, numeric actions, and verification of the resulting geometry across furniture, industrial components, and real fracture fragments. Fixed part geometry and scale make successful reconstruction assessable, while global alignment and equivalent-part matching accommodate irrelevant overall pose and interchangeable components without independently repositioning parts during scoring. Paired PartNet reference conditions and shared initial states make within-benchmark comparisons interpretable. The setting also complements specialized multimodal pose prediction such as the already-cited AssemLM, which uses a dedicated point-cloud encoder. This combination of task definition, interaction, and cross-domain evaluation is the main reason I support acceptance.
-
The empirical analysis provides useful evidence about both successful assembly and remaining failure mechanisms. The paper separates partial part placement from complete reconstruction and examines how reference information, intermediate edits, and residual errors relate to the outcomes. The appendix preserves object pairing in uncertainty estimates and explicitly states that the Astra–Fable difference is not significant after the reported multiple-comparison correction. Particularly useful is the matched four-way experiment: applying GARF after the agent reduces PA from 91.67% to 87.00%, whereas GARF followed by the agent reaches 93.00% on the same 150 two-fragment objects, with improvements and regressions reported per object. The negative composition result gives concrete guidance for future refinement methods. Together with the larger single-agent evaluations in Table A-1, these analyses add research value beyond the common benchmark's aggregate scores.
Critical weaknesses
-
The comparisons with specialized methods do not sufficiently establish performance advantages under a common evaluation protocol. Section 4.3 and Tables 2–5 compare Astra with results obtained under the specialized methods' reported settings. Appendix A.3 explicitly states that the Fantastic Breaks alignment and sampled inputs do not constitute a matched reproduction of the published runs, and that the IKEA alternative success metric, , uses each method's own normalization. Consequently, the same numerical distance threshold need not represent the same geometric tolerance. For the furniture and industrial comparisons, it is not sufficiently clear that normalization, point sampling, global registration, equivalent-part assignment, and initialization differences have been reconciled. This is especially consequential for interpreting Table 3's 100% and the PA/SR advantages emphasized in Section 4.3. These results demonstrate capability, but the claim of superiority to specialized methods needs at least a representative comparison with a shared evaluator or narrower wording. This concern does not invalidate the common eight-system comparison or the internally matched experiment in Table 6.
-
Visual-input validity remains unresolved for part of the low-performing system comparison. Appendix C.2 documents a DeepSeek run that reports receiving no viewable image despite image payloads in the tool records; whether those images actually reached the model through its provider remains unresolved. Qwen and DeepSeek also have substantial execution failures and timeouts. Counting such failures is appropriate for an end-to-end system benchmark, and the paper correctly identifies the evaluated units as model–harness combinations. Nevertheless, successful image delivery is a prerequisite for interpreting the results as evidence about visual assembly capability. Identical tool schemas do not establish that prerequisite across provider adapters. An end-to-end check of image consumption and basic observation/action functionality is needed to determine whether the low scores reflect assembly limitations, integration faults, or both. This limits the interpretation of the reported gap involving these systems; it is not evidence that image delivery failed in every run.
Questions for authors
-
Clarification and targeted validation of the specialized-method comparisons. Which rows in Tables 2–4 share Astra's object set, reference condition, normalization, point sampling, global alignment, equivalent-part matching, and success threshold, and which retain a different protocol? In particular, how should Table 3's values be compared when normalization is method-specific? For at least one representative furniture or industrial baseline with available predictions or an implementation, can you report results using a shared evaluator and clearly specified input/initialization conditions? Whether the reported PA/SR advantage persists would materially affect my assessment of the comparative evidence.
-
Clarification and targeted validation of provider image delivery. Can you establish whether Qwen and DeepSeek receive usable camera images through the evaluated adapters, including the unresolved case in Appendix C.2? Please describe an end-to-end check that demonstrates image consumption and a simple observation–action–observation cycle, and quantify any affected runs identifiable from the records. If a delivery or adapter problem is found, results on a fixed, representative subset after correction would help determine whether the reported system gap persists. The answer would materially affect my confidence in the interpretation of the low-performing systems.
3 comments
Dear Reviewer U9wr,
Thank you again for your careful review and for raising the questions about evaluation comparability and image delivery. In our response, we clarified the evaluation protocols used for comparisons with specialized methods (W1/Q1) and reported end-to-end image-delivery checks and assembly replays for DeepSeek and Qwen (W2/Q2).
We hope these clarifications are helpful. If either point would benefit from further discussion, we would be happy to follow up before the discussion closes.
Best regards, The authors of Paper 35762
Thanks for the additional experiments to clarify my concerns. I would like to maintain my score.
Response to Reviewer U9wr
W1 / Q1. Evaluation comparability
We clarify that the differences in evaluation implementation concern three aspects: point sampling, global alignment, and scale normalization. For the matched comparisons reported in Tables 2–4, we retain the same PA/SR definitions, success thresholds, and equivalent-part matching principles. These differences arise from evaluating assemblies from different data sources in a common interactive environment. Below, we explain each difference and its effect on the reported results.
Point sampling
Geometry and pre-sampled point clouds are not provided uniformly across sources. We therefore generate evaluation point clouds from meshes using a common procedure and fix the random seed for reproducibility. On five IKEA-Manual objects with fixed final assemblies, we resampled and rescored the results across three seeds. Mean SCD ranged from 3.558×10−4 to 3.664×10−4, with a maximum relative change of approximately 3.0 from the original sampling seed, while PA, SR, and SRᵃ remained unchanged for every object.
Global alignment
Existing methods typically predict poses in a specified target coordinate frame, whereas AssemblyWorld allows agents to assemble objects in a free coordinate frame. We therefore apply a single rigid transformation to the entire assembly. This alignment removes only the arbitrary global coordinate frame of an assembly. It does not independently reposition parts or correct their relative placement errors. In the two-fragment comparison with GARF, both configurations use the same fixed-reference-part anchor alignment.
Normalization
We normalize scale during offline evaluation to apply a consistent scoring convention across different sources. Applying the normalization convention described by AssemLM to our predictions leaves SRᵃ unchanged at 100 across all 102 IKEA objects. The maximum SCD is 0.00728, well below the shared threshold of 0.02. Thus, the normalization difference does not explain the 100 SRᵃ reported in Table 3.
The sampling and normalization checks therefore preserve the corresponding reported success metrics, while global alignment accounts for the arbitrary overall pose permitted by the interactive setting without changing the assembled structure.
W2 / Q2. Image delivery and assembly replays
We conducted five end-to-end image checks each for DeepSeek and Qwen, covering the initial reference image, manual images, and camera screenshots in a cycle of observation, action, and further observation. Both systems passed all checks, and the scene records confirmed a valid part displacement. Each system then replayed five fixed assembly cases. DeepSeek had PA and SR both equal to zero in all five cases and made no part-pose edits, remaining mainly in tool and scene queries. For Qwen, one run terminated normally and four timed out. All five obtained PA=0 and SR=0. These results show that, in the current configuration, usable images alone do not ensure effective assembly.