- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper is a benchmark paper to measure the capability of existing VLMs to correctly assemble parts into 3D objects. Compared to existing evaluation, they propose more metrics that evaluate part correctness, rather than full-object metrics like IoU. They evaluate state-of-the-art VLMs like Fable, Sol and Kimi on this benchmark.
Critical strengths
The paper provides additional metrics to evaluate part assembly that is not evaluated previously. The authors curate a benchmark of 1195 manually annotated objects selected from 82717 Blendkit models across 12 category groups, which is effort reusable by the community.
Critical weaknesses
The benchmark annotation process is VLM-assisted and might introduce model bias. While it is nice to comprehensively evaluate part metrics in addition to object metrics for VLM object assembly, I do not find the contribution substantial enough for a conference paper. The paper assumes the parts already exist and just need assembling, which is a big assumption that is not generally met in the real world. Given this, why is this benchmark practically relevant?
Questions for authors
- Could you comment on why this benchmark is significant - since the benchmark setting (assuming parts already exist) is quite far from practical use, does it expose any good insights on VLM development?
- Could you comment on the potential bias introduced by the VLM-assisted annotation process?
9 comments
[W1 & Q1] Motivation for Pre-Existing Part Specifications and Five Insights for VLM Development [part 1/3]
Thank you for raising this fundamental question regarding the practical relevance and diagnostic value of our task formulation. In our initial manuscript, Section 3 focused primarily on metric definitions without sufficiently motivating upfront why tasks supply a pre-existing part inventory and part-level reference views—which could make the benchmark appear as if ready-made 3D parts already exist and only need pose assembly, whereas in fact source 3D meshes are strictly hidden and models must synthesize every component's 3D geometry from scratch and assemble it in place via Blender code. To address your concern directly: PartGround-Bench intentionally adopts this controlled specification-driven setup both to mirror real-world CAD/graphics production workflows built from engineering bills of materials (BOMs) and callout sheets and to decouple open-ended semantic decomposition ambiguity from 3D geometric parameterization and in-situ spatial assembly, enabling deterministic part-level scoring without post-hoc segmentation noise or per-part pose realignment. Crucially, holding the target part specification fixed exposes five actionable insights for VLM development that whole-object metrics obscure: even when program execution (94.2%–97.8% single-call; 100.0% agents) and supplied part-name instantiation (0.982–0.986 Name Recall) are near saturation, frontier VLMs remain severely bottlenecked on in-situ 3D part grounding ([email protected] of 0.060–0.183 single-call and 0.021–0.168 agents) and exhibit systematic trade-offs across visual part conditioning, reasoning-effort scaling, and organic freeform geometry (summarized self-contained in the table below).
1. Why does PartGround-Bench intentionally adopt a controlled setting where the part decomposition and part-level reference views already exist? (Three Core Motivations)
As you rightly observe, PartGround-Bench constructs tasks from pre-segmented 3D assets and supplies the evaluated VLM with three overall views (front, side, top) alongside a pre-existing semantic part-name inventory—accompanied either by textual material and dimension hints (Textual Part Guidance) or by rendered unoccluded 2D close-ups and whole-object location insets for each part (Visual Part Guidance)—asking the model to construct each specified component's 3D geometry from scratch and assemble it in place via Blender Python code, with source 3D meshes hidden rather than imported (Sections 3.1 and 3.2 of the revised PDF). The benchmark therefore evaluates specification-driven procedural 3D reconstruction and assembly diagnosis rather than open-world part discovery. This controlled design is essential for three reasons:
- First, decoupling open-ended decomposition ambiguity from 3D construction and spatial assembly to enable noise-free deterministic evaluation:
Part granularity in 3D objects is inherently non-unique (e.g., a chair can be validly decomposed into 4 major assembly groups or 20 individual slats and fasteners). Without a supplied target part specification, different models produce incompatible part counts and boundaries; part-level scoring then typically relies either on reference-free black-box neural evaluators (
Hi3DEval) or, for part-level scoring inP3D-Bench, on post-hoc LLM code splitting followed by Hungarian matching and independent per-part pose realignment, which translates misplaced parts back to the ground-truth centroid and erases assembly placement errors from part shape scores. Supplying the pre-existing part inventory establishes exact one-to-one semantic correspondence ("Names determine the correspondences specified by the task, rather than selecting pairs to maximize geometric similarity"). Under a single shared whole-object rigid transformation , this allows PartGround-Bench to jointly evaluate component geometry and in-situ assembled placement across 1,195 objects without post-hoc segmentation noise or learned judges (Sections 1 and 3.2 of the revised PDF).
[W1 & Q1] Motivation for Pre-Existing Part Specifications and Five Insights for VLM Development [part 2/3]
- Second, establishing a controlled diagnostic setting that reduces ambiguity in part discovery and inter-part occlusion: In unconstrained single-image reconstruction, when a model fails, it is difficult to tell whether it failed to infer what parts exist (semantic decomposition and whole-object occlusion reasoning) or failed to parameterize and position those parts in 3D space (geometric synthesis and spatial assembly). By explicitly providing the part names, dimension hints, or isolated part close-ups with context insets, PartGround-Bench reduces uncertainty about the required part inventory and inter-part occlusion (while the model must still infer 3D shape, depth, and unseen surfaces from limited 2D observations). Showing that frontier VLMs still struggle to construct and assemble components accurately even when the part inventory and per-part reference evidence are explicitly supplied demonstrates that—beyond open-world part discovery—3D geometric parameterization and cross-part coordinate grounding remain critical bottlenecks in their own right.
- Third, matching specification-driven 3D production workflows: While diffusion models synthesize monolithic meshes, graphics, CAD, and robotics pipelines require programs that "expose named components, dimensions, and spatial relationships for targeted edits", material assignment, and articulation. In real production workflows, modeling briefs, engineering bills of materials (BOMs), and callout sheets routinely specify the required component list and reference views upfront (Sections 1 and 2.1 of the revised PDF).
We have highlighted these three design motivations in Sections 1 and 3.2 of the revised PDF (in blue).
2. Five actionable insights for VLM development clearly exposed by supplying the part inventory and part-level views:
- Switching from Textual to Visual Part Guidance improves iterative agents across
IoU,SQ,Part-F1, material fidelity, , andContactF, but is accompanied by lower mean relative layout across all 9 single-call configurations: This contrast is directly enabled by our controlled part-guidance setup. Under Visual Part Guidance, textual material and approximate dimension hints are replaced with unoccluded close-ups and location insets for every part. Across all 5 closed-loop agent configurations, this visual part evidence consistently improves whole-objectIoU, part overlapSQ,Part-F1(+0.009to+0.068), material fidelity, relative layout (by up to+0.087), and symmetric contact consistencyContactF(computed via bilateral 1,500-point surface sampling,contact_v2; by up to+0.062). In stark contrast, across all 9 single-call VLM configurations, while switching to visual part guidance improves local part overlapSQin 7 of 9 cases, mean global relative layout decreases in 9 out of 9 configurations (-0.001to-0.037on successful outputs). When textual material and approximate dimension hints are replaced with multiple local part close-ups and context insets, single-call models without execution or render feedback improve local part overlap in most configurations yet consistently lose global cross-part layout coherence—consistent with the hypothesis that integrating multiple local views without iterative feedback makes it harder to maintain a unified global 3D coordinate frame (Section 4.3 and Table 1 of the revised PDF). - When the part inventory is supplied, code execution and part instantiation are near saturation, highlighting 3D geometric synthesis and spatial grounding as a major bottleneck:
Because target part names are provided as input, single-call models achieve
94.2%–97.8%program execution success across all 1,195 objects, and all five agent configurations achieve100.0%execution success with0.982–0.986Name Recallon the 185-object subset. Yet in-situ[email protected]remains low (0.060–0.183for single-call;0.021–0.168for agents) and symmetricContactF-AUCreaches at most0.553among no-skill agents. High execution rates and near-complete coverage of supplied part names alongside lowPart-F1show that even when syntax errors and inventory omissions are largely removed, translating 2D references into accurate 3D part geometry and assembled coordinates remains a major bottleneck (Section 4.2, Appendix B.2, and Tables 1 and 4 of the revised PDF).
Thank you for your constructive review, and for recognizing the community value of curating 1,195 manually annotated multi-part 3D objects from 82,717 Blendkit models across 12 category groups alongside part- and assembly-level evaluation metrics. You rightly point out a key premise and scope boundary of our benchmark setting: PartGround-Bench indeed assumes a pre-existing part decomposition (evaluating specification-driven procedural 3D reconstruction and assembly rather than open-world part discovery), supplying evaluated VLMs with the target part-name inventory alongside textual part hints or rendered 2D part close-ups and location insets so the model can construct and assemble those components into a complete 3D object via Blender code. Below, we first explain why this controlled setup is both methodologically essential and practically relevant, then highlight five concrete insights for VLM development clearly exposed by this controlled setting, and finally detail the bias-mitigation safeguards and cross-family empirical checks in our VLM-assisted preprocessing and human-verified annotation pipeline. All corresponding clarifications have been incorporated into the revised PDF (highlighted in blue).