acceptodds
Under review as a conference paper at ICLR 2027

PartGround: Benchmarking, Analyzing, and Improving VLM-Based 3D Generation

Abstract

Vision-Language Models (VLMs) can generate editable 3D assets through code, complementing diffusion-based methods. Whole-object similarity alone cannot establish component, material, and relational fidelity in an assembled asset. To address this evaluation need, we introduce PartGround-Bench for reconstruction from reference images and supplied part specifications, with 1,195 manually annotated objects selected from 82,717 Blendkit models across 12 category groups. Fixed part correspondences and shared whole-object alignment enable direct scoring of overall shape, part geometry and completeness, per-part materials, and spatial relationships, preserving placement errors. We evaluate nine single-call configurations on the full benchmark and five agent configurations on a subset. Under Visual Part Guidance, two agents achieve similar mean IoU (0.370 and 0.372) but different mean Part-F1 (0.149 and 0.168). Higher reasoning effort accompanies better single-call geometry scores, without uniformly improving materials or reducing inter-part penetration. We further evaluate a reusable multi-agent modeling skill with shared planning and independent review, improving overall and part geometry, layout, and contact consistency across three agent configurations. For Kimi-K3 (max) under Visual Part Guidance, our relative gains are 5.9% in whole-object IoU, 17.4% in Part-F1, 3.2% in layout \(\DeltaAUC\), and 8.2% in ContactF-AUC, demonstrating the effectiveness of our multi-agent modeling skill in improving geometric and relational fidelity.

Primary area:
datasets and benchmarks
Keywords:
Procedural 3D Modeling Benchmark, Vision-Language Models, Multi-Agent workflow

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_j4CHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper is a benchmark paper to measure the capability of existing VLMs to correctly assemble parts into 3D objects. Compared to existing evaluation, they propose more metrics that evaluate part correctness, rather than full-object metrics like IoU. They evaluate state-of-the-art VLMs like Fable, Sol and Kimi on this benchmark.

Critical strengths

The paper provides additional metrics to evaluate part assembly that is not evaluated previously. The authors curate a benchmark of 1195 manually annotated objects selected from 82717 Blendkit models across 12 category groups, which is effort reusable by the community.

Critical weaknesses

The benchmark annotation process is VLM-assisted and might introduce model bias. While it is nice to comprehensively evaluate part metrics in addition to object metrics for VLM object assembly, I do not find the contribution substantial enough for a conference paper. The paper assumes the parts already exist and just need assembling, which is a big assumption that is not generally met in the real world. Given this, why is this benchmark practically relevant?

Questions for authors
  1. Could you comment on why this benchmark is significant - since the benchmark setting (assuming parts already exist) is quite far from practical use, does it expose any good insights on VLM development?
  2. Could you comment on the potential bias introduced by the VLM-assisted annotation process?
9 comments

Thank you for your constructive review, and for recognizing the community value of curating 1,195 manually annotated multi-part 3D objects from 82,717 Blendkit models across 12 category groups alongside part- and assembly-level evaluation metrics. You rightly point out a key premise and scope boundary of our benchmark setting: PartGround-Bench indeed assumes a pre-existing part decomposition (evaluating specification-driven procedural 3D reconstruction and assembly rather than open-world part discovery), supplying evaluated VLMs with the target part-name inventory alongside textual part hints or rendered 2D part close-ups and location insets so the model can construct and assemble those components into a complete 3D object via Blender code. Below, we first explain why this controlled setup is both methodologically essential and practically relevant, then highlight five concrete insights for VLM development clearly exposed by this controlled setting, and finally detail the bias-mitigation safeguards and cross-family empirical checks in our VLM-assisted preprocessing and human-verified annotation pipeline. All corresponding clarifications have been incorporated into the revised PDF (highlighted in blue).

[W1 & Q1] Motivation for Pre-Existing Part Specifications and Five Insights for VLM Development [part 1/3]

Thank you for raising this fundamental question regarding the practical relevance and diagnostic value of our task formulation. In our initial manuscript, Section 3 focused primarily on metric definitions without sufficiently motivating upfront why tasks supply a pre-existing part inventory and part-level reference views—which could make the benchmark appear as if ready-made 3D parts already exist and only need pose assembly, whereas in fact source 3D meshes are strictly hidden and models must synthesize every component's 3D geometry from scratch and assemble it in place via Blender code. To address your concern directly: PartGround-Bench intentionally adopts this controlled specification-driven setup both to mirror real-world CAD/graphics production workflows built from engineering bills of materials (BOMs) and callout sheets and to decouple open-ended semantic decomposition ambiguity from 3D geometric parameterization and in-situ spatial assembly, enabling deterministic part-level scoring without post-hoc segmentation noise or per-part pose realignment. Crucially, holding the target part specification fixed exposes five actionable insights for VLM development that whole-object metrics obscure: even when program execution (94.2%–97.8% single-call; 100.0% agents) and supplied part-name instantiation (0.982–0.986 Name Recall) are near saturation, frontier VLMs remain severely bottlenecked on in-situ 3D part grounding ([email protected] of 0.060–0.183 single-call and 0.021–0.168 agents) and exhibit systematic trade-offs across visual part conditioning, reasoning-effort scaling, and organic freeform geometry (summarized self-contained in the table below).

1. Why does PartGround-Bench intentionally adopt a controlled setting where the part decomposition and part-level reference views already exist? (Three Core Motivations)

As you rightly observe, PartGround-Bench constructs tasks from pre-segmented 3D assets and supplies the evaluated VLM with three overall views (front, side, top) alongside a pre-existing semantic part-name inventory—accompanied either by textual material and dimension hints (Textual Part Guidance) or by rendered unoccluded 2D close-ups and whole-object location insets for each part (Visual Part Guidance)—asking the model to construct each specified component's 3D geometry from scratch and assemble it in place via Blender Python code, with source 3D meshes hidden rather than imported (Sections 3.1 and 3.2 of the revised PDF). The benchmark therefore evaluates specification-driven procedural 3D reconstruction and assembly diagnosis rather than open-world part discovery. This controlled design is essential for three reasons:

  • First, decoupling open-ended decomposition ambiguity from 3D construction and spatial assembly to enable noise-free deterministic evaluation: Part granularity in 3D objects is inherently non-unique (e.g., a chair can be validly decomposed into 4 major assembly groups or 20 individual slats and fasteners). Without a supplied target part specification, different models produce incompatible part counts and boundaries; part-level scoring then typically relies either on reference-free black-box neural evaluators (Hi3DEval) or, for part-level scoring in P3D-Bench, on post-hoc LLM code splitting followed by Hungarian matching and independent per-part pose realignment, which translates misplaced parts back to the ground-truth centroid and erases assembly placement errors from part shape scores. Supplying the pre-existing part inventory establishes exact one-to-one semantic correspondence ("Names determine the correspondences specified by the task, rather than selecting pairs to maximize geometric similarity"). Under a single shared whole-object rigid transformation TT, this allows PartGround-Bench to jointly evaluate component geometry and in-situ assembled placement across 1,195 objects without post-hoc segmentation noise or learned judges (Sections 1 and 3.2 of the revised PDF).

[W1 & Q1] Motivation for Pre-Existing Part Specifications and Five Insights for VLM Development [part 2/3]

  • Second, establishing a controlled diagnostic setting that reduces ambiguity in part discovery and inter-part occlusion: In unconstrained single-image reconstruction, when a model fails, it is difficult to tell whether it failed to infer what parts exist (semantic decomposition and whole-object occlusion reasoning) or failed to parameterize and position those parts in 3D space (geometric synthesis and spatial assembly). By explicitly providing the part names, dimension hints, or isolated part close-ups with context insets, PartGround-Bench reduces uncertainty about the required part inventory and inter-part occlusion (while the model must still infer 3D shape, depth, and unseen surfaces from limited 2D observations). Showing that frontier VLMs still struggle to construct and assemble components accurately even when the part inventory and per-part reference evidence are explicitly supplied demonstrates that—beyond open-world part discovery—3D geometric parameterization and cross-part coordinate grounding remain critical bottlenecks in their own right.
  • Third, matching specification-driven 3D production workflows: While diffusion models synthesize monolithic meshes, graphics, CAD, and robotics pipelines require programs that "expose named components, dimensions, and spatial relationships for targeted edits", material assignment, and articulation. In real production workflows, modeling briefs, engineering bills of materials (BOMs), and callout sheets routinely specify the required component list and reference views upfront (Sections 1 and 2.1 of the revised PDF).

We have highlighted these three design motivations in Sections 1 and 3.2 of the revised PDF (in blue).

2. Five actionable insights for VLM development clearly exposed by supplying the part inventory and part-level views:

  1. Switching from Textual to Visual Part Guidance improves iterative agents across IoU, SQ, Part-F1, material fidelity, ΔAUC\Delta\mathrm{AUC}, and ContactF, but is accompanied by lower mean relative layout ΔAUC\Delta\mathrm{AUC} across all 9 single-call configurations: This contrast is directly enabled by our controlled part-guidance setup. Under Visual Part Guidance, textual material and approximate dimension hints are replaced with unoccluded close-ups and location insets for every part. Across all 5 closed-loop agent configurations, this visual part evidence consistently improves whole-object IoU, part overlap SQ, Part-F1 (+0.009 to +0.068), material fidelity, relative layout ΔAUC\Delta\mathrm{AUC} (by up to +0.087), and symmetric contact consistency ContactF (computed via bilateral 1,500-point surface sampling, contact_v2; by up to +0.062). In stark contrast, across all 9 single-call VLM configurations, while switching to visual part guidance improves local part overlap SQ in 7 of 9 cases, mean global relative layout ΔAUC\Delta\mathrm{AUC} decreases in 9 out of 9 configurations (-0.001 to -0.037 on successful outputs). When textual material and approximate dimension hints are replaced with multiple local part close-ups and context insets, single-call models without execution or render feedback improve local part overlap in most configurations yet consistently lose global cross-part layout coherence—consistent with the hypothesis that integrating multiple local views without iterative feedback makes it harder to maintain a unified global 3D coordinate frame (Section 4.3 and Table 1 of the revised PDF).
  2. When the part inventory is supplied, code execution and part instantiation are near saturation, highlighting 3D geometric synthesis and spatial grounding as a major bottleneck: Because target part names are provided as input, single-call models achieve 94.2%–97.8% program execution success across all 1,195 objects, and all five agent configurations achieve 100.0% execution success with 0.982–0.986 Name Recall on the 185-object subset. Yet in-situ [email protected] remains low (0.060–0.183 for single-call; 0.021–0.168 for agents) and symmetric ContactF-AUC reaches at most 0.553 among no-skill agents. High execution rates and near-complete coverage of supplied part names alongside low Part-F1 show that even when syntax errors and inventory omissions are largely removed, translating 2D references into accurate 3D part geometry and assembled coordinates remains a major bottleneck (Section 4.2, Appendix B.2, and Tables 1 and 4 of the revised PDF).
Reviewer_b16DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PartGround-Bench, a benchmark of 1,195 manually annotated 3D objects across 12 categories for evaluating VLM-based reconstruction from reference images and supplied part specifications. Its central design uses fixed semantic part correspondences and shared whole-object alignment to preserve both shape and placement errors, complemented by material and inter-part relationship metrics. The authors evaluate nine single-call configurations on the full benchmark and five agent configurations on a 185-object subset. They also introduce a multi-agent modeling workflow with shared planning and independent review. Results illustrate that similar whole-object scores can conceal differences in part fidelity, while workflow improvements incur additional computational costs.

Critical strengths
  1. Fixed semantic correspondences and shared whole-object alignment prevent independent part alignment from hiding assembly errors. Equal part weighting and completeness measures also give small components an explicit contribution. These choices align well with the objective of determining whether specified components are correctly realized in the completed asset.
  2. Two agents achieve nearly identical whole-object IoU scores of 0.370 and 0.372, yet their Part-F1 scores differ at 0.149 and 0.168. This provides concrete evidence that aggregate geometric similarity alone does not fully characterize reconstruction quality, supporting the benchmark’s potential usefulness.
Critical weaknesses
  1. Appendix A.4 states that the material metric uses the complete CIEDE2000 definition. However, the supplementary materials.py implements delta_e_00 as an explicitly documented simplified Lab discrepancy and calls it directly in material scoring. It is therefore unclear which implementation produced the reported results. Additionally, ContactF-AUC uses surface sampling for reference geometry but vertex subsampling for predictions, potentially introducing sensitivity to tessellation and vertex distribution. Although this limitation is acknowledged, its effect on rankings and reported improvements is not established. For a benchmark-centered contribution, these issues directly affect confidence in the evaluation.
  2. The independent contribution of the multi-agent workflow is not sufficiently isolated. In Table 3, the workflow increases token usage by approximately 36–42%, while the absolute Part-F1 improvements for Claude and GPT are only 0.003 and 0.002, respectively. No compute-matched controls, component ablations, or confidence intervals are reported to establish whether these gains reflect the proposed coordination mechanisms rather than additional computation or run-to-run variability. The authors explicitly acknowledge this attribution limitation, but it still weakens the methodological contribution.
Questions for authors
  1. Were the material scores in Tables 1 and 3 computed using complete CIEDE2000 or the simplified implementation in the supplementary code? After reconciling this discrepancy, do the model rankings and material-related conclusions remain unchanged?

  2. How does ContactF-AUC behave when the reference geometry is used directly as the prediction, or when only tessellation and vertex ordering are changed? Do the principal conclusions hold when identical contact-detection procedures are used for reference and predicted geometry?

  3. Does the multi-agent workflow still outperform the single-agent baseline under matched token or computational budgets? Can the authors provide paired confidence intervals and ablations separating the contributions of shared planning, parallel construction, and independent review?

15 comments

Thank you for your thorough, line-by-line examination of our metric definitions, appendices, and supplementary code (eval/materials.py and contact graph sampling), and for recognizing the value of fixed semantic correspondences, shared whole-object alignment, and our empirical finding that similar whole-object IoU scores can conceal substantial differences in part fidelity. Below, we address each of your concerns through analytical error-propagation proofs and controlled re-evaluations, and we have updated both the revised PDF (highlighted in blue) and the supplementary evaluation code accordingly.

AuthorsOpenReview ↗

[W1a & Q1] Material Fidelity Metric: Complete CIEDE2000 Proof and 31-Configuration Re-Evaluation [part 1/3]

Thank you for catching this discrepancy between Appendix A.4 and the supplementary eval/materials.py. This mismatch arose because an earlier compatibility routine (delta_e_00 documented as a simplified Lab discrepancy) was inadvertently left in eval/materials.py when packaging the supplementary snapshot, rather than being updated to the production implementation used for the paper's tables. To answer your question directly: all material scores (Mat., BmaterialB_{\mathrm{material}}) reported in Tables 1 and 3 were computed using the complete CIEDE2000 formula defined in Appendix A.4 (Sharma et al., 2005). We have updated the supplementary eval/materials.py to this exact complete CIEDE2000 implementation, and—combining conditional error-propagation analysis with a full re-evaluation across all 31 configurations in Tables 1 and 3—we confirm that empirical score differences between the two versions are negligible (0.000–0.003 on 30/31 configurations and 0.005 on 1 configuration, Spearman ρ=0.995\rho = 0.995 across all 31 displayed values and ρ=0.993\rho = 0.993 across the 28 Table 1 rows, or ρ=0.996\rho = 0.996 unrounded), 100% of within-guidance agent rankings in Table 1 and multi-agent rankings in Table 3 remain unchanged (ρ=1.000\rho = 1.000), Table 3 skill material changes remain identical in sign and within 0.001 in magnitude (+0.003 / -0.002 / +0.017 vs. +0.003 / -0.002 / +0.016), and all material-related conclusions hold.

1. Conditional error-propagation analysis: how truncation and weighting in BmaterialB_{\mathrm{material}} compress ΔE00\Delta E_{00} discrepancies:

Writing out the complete CIEDE2000 color difference with unit parametric factors (dcfulld_c^{\mathrm{full}}, Sharma et al., 2005; Appendix A.4) side by side with the simplified routine in the originally submitted supplementary snapshot (dcsimpd_c^{\mathrm{simp}} in eval/materials.py, retained as delta_e_00_simplified for historical comparison),

dcfull=(ΔL′SL)2+(ΔC′SC)2+(ΔH′SH)2+RT(ΔC′SC)(ΔH′SH),dcsimp=(ΔLSL)2+(ΔCSC)2+ΔH2SH2+RTsimp(ΔCSC)ΔH2SH,d_c^{\mathrm{full}} = \sqrt{\left(\frac{\Delta L'}{S_L}\right)^2 + \left(\frac{\Delta C'}{S_C}\right)^2 + \left(\frac{\Delta H'}{S_H}\right)^2 + R_T\left(\frac{\Delta C'}{S_C}\right)\left(\frac{\Delta H'}{S_H}\right)}, \qquad d_c^{\mathrm{simp}} = \sqrt{\left(\frac{\Delta L}{S_L}\right)^2 + \left(\frac{\Delta C}{S_C}\right)^2 + \frac{\Delta H^2}{S_H^2} + R_T^{\mathrm{simp}}\left(\frac{\Delta C}{S_C}\right)\frac{\sqrt{\Delta H^2}}{S_H}},

shows directly how formula-level differences are conditionally truncated and damped when mapped to BmaterialB_{\mathrm{material}}:

(1a) Shared primary terms and the Law-of-Cosines identity for squared hue difference. Both functions share the exact same sRGB →\to D65 CIELAB conversion, the lightness term ∣ΔL′∣/SL|\Delta L'| / S_L (with ∣ΔL′∣=∣L2∗−L1∗∣=∣ΔL∣|\Delta L'| = |L_2^{\ast} - L_1^{\ast}| = |\Delta L| and SL=1+0.015(Lˉ−50)220+(Lˉ−50)2S_L = 1 + \frac{0.015(\bar{L}-50)^2}{\sqrt{20+(\bar{L}-50)^2}}), and the functional forms of SC=1+0.045CˉS_C = 1 + 0.045\bar{C}, SH=1+0.015CˉT(Hˉ)S_H = 1 + 0.015\bar{C}T(\bar{H}), and the 4th-order cosine harmonic T(Hˉ)=1−0.17cos⁡(Hˉ−30∘)+0.24cos⁡(2Hˉ)+0.32cos⁡(3Hˉ+6∘)−0.20cos⁡(4Hˉ−63∘)T(\bar{H}) = 1 - 0.17\cos(\bar{H}-30^\circ) + 0.24\cos(2\bar{H}) + 0.32\cos(3\bar{H}+6^\circ) - 0.20\cos(4\bar{H}-63^\circ). While the simplified snapshot function writes dH2 = max(0.0, da*da + db*db - dC*dC), the Law of Cosines in the unprimed (a,b)(a,b) plane gives

Δa2+Δb2−ΔC2=(C12+C22−2C1C2cos⁡Δh)−(C1−C2)2=2C1C2(1−cos⁡Δh)=4C1C2sin⁡2 ⁣(Δh2)≡(ΔH)2,\Delta a^2 + \Delta b^2 - \Delta C^2 = \left(C_1^2 + C_2^2 - 2C_1 C_2\cos\Delta h\right) - (C_1 - C_2)^2 = 2C_1 C_2(1-\cos\Delta h) = 4C_1 C_2\sin^2\!\left(\frac{\Delta h}{2}\right) \equiv (\Delta H)^2,

so the squared hue term itself is algebraically identical to (2C1C2sin⁡(Δh/2))2\big(2\sqrt{C_1 C_2}\sin(\Delta h/2)\big)^2 in the unprimed coordinates. Alongside standard branch handling of mean hue Hˉ\bar{H} across 0∘0^\circ and zero-chroma pairs, the two color-difference formulas differ in only three places:

  • Low-chroma a∗a^{\ast}-axis scaling: Complete CIEDE2000 applies ai′=(1+G)ai∗a_i' = (1+G)a_i^{\ast} with G=0.5(1−Cˉab7/(Cˉab7+257))∈[0,0.5]G = 0.5\big(1 - \sqrt{\bar{C}_{ab}^7/(\bar{C}_{ab}^7+25^7)}\big) \in [0, 0.5], whereas the simplified function sets G=0G = 0.
  • Chroma ratio in the blue-region rotation factor RTR_T: In RT=−2Cˉ′7/(Cˉ′7+257)sin⁡(2Δθ)R_T = -2\sqrt{\bar{C}'^7/(\bar{C}'^7+25^7)}\sin(2\Delta\theta) (where Δθ=30∘exp⁡(−[(Hˉ−275∘)/25∘]2)\Delta\theta = 30^\circ\exp(-[(\bar{H}-275^\circ)/25^\circ]^2) decays rapidly outside the blue region around Hˉ≈275∘\bar{H}\approx 275^\circ), the simplified RTsimpR_T^{\mathrm{simp}} uses the quadratic chroma ratio C1C2/(C1C2+252)\sqrt{C_1 C_2/(C_1 C_2+25^2)} instead of the 7th-power ratio Cˉ′7/(Cˉ′7+257)\sqrt{\bar{C}'^7/(\bar{C}'^7+25^7)}.
  • Sign convention in the rotation cross term: Complete CIEDE2000 uses ΔC′=C2′−C1′\Delta C' = C_2' - C_1' and the signed hue difference ΔH′=2C1′C2′sin⁡(Δh′/2)\Delta H' = 2\sqrt{C_1' C_2'}\sin(\Delta h'/2), whereas the simplified function uses ΔC=C1−C2\Delta C = C_1 - C_2 and the nonnegative magnitude ΔH2\sqrt{\Delta H^2}.
AuthorsOpenReview ↗

[W1a & Q1] Material Fidelity Metric: Complete CIEDE2000 Proof and 31-Configuration Re-Evaluation [part 2/3]

(1b) Complete definition and conditional contraction bound in BmaterialB_{\mathrm{material}}.

Let Mmat\mathcal{M}_{\mathrm{mat}} denote the set of semantically matched part pairs with valid base color, roughness raXr_a^X, and metallicity μaX∈[0,1]\mu_a^X \in [0,1] on both the predicted (X=PX=P) and reference (X=GX=G) sides. Task-level material fidelity averages the per-part scores equally (Section 3.3, Eq. 2 and Appendix A.4, Eq. 6).

Bmaterial=1∣Mmat∣∑a∈MmatBmaterial(a).B_{\mathrm{material}} = \frac{1}{|\mathcal{M}_{\mathrm{mat}}|} \sum_{a \in \mathcal{M}_{\mathrm{mat}}} B_{\mathrm{material}}(a).

Bmaterial(a)=1−(0.5eca+0.3era+0.2eμa).B_{\mathrm{material}}(a) = 1 - (0.5 e_c^a + 0.3 e_r^a + 0.2 e_\mu^a).

The normalized color and metallicity errors are eca=min⁡(dca/5,1)e_c^a = \min(d_c^a/5, 1) and eμa=∣μaP−μaG∣e_\mu^a = |\mu_a^P - \mu_a^G|. The normalized roughness error uses rˉaX=clip⁡(raX,0.02,1)\bar{r}_a^X = \operatorname{clip}(r_a^X, 0.02, 1).

era=min⁡(∣log⁡2(rˉaP/rˉaG)∣2,1).e_r^a = \min\left(\frac{|\log_2(\bar{r}_a^P / \bar{r}_a^G)|}{2}, 1\right).

With the matched parts and valid attributes fixed, the roughness and metallicity terms (50% of the total weight) are identical in the full and simplified versions. For a given pair aa, let ecfulle_c^{\mathrm{full}} and ecsimpe_c^{\mathrm{simp}} be the normalized color errors computed from dcfulld_c^{\mathrm{full}} and dcsimpd_c^{\mathrm{simp}}, respectively. The score difference is exactly

= \tfrac{1}{2}|e_c^{\mathrm{full}} - e_c^{\mathrm{simp}}|.$$ The normalized color error is capped at 1 and is $0.2$-Lipschitz in the color distance. Thus the per-part score difference satisfies $$|B_{\mathrm{material}}^{\mathrm{full}}(a) - B_{\mathrm{material}}^{\mathrm{simp}}(a)| \le \min\left(\tfrac{1}{2}, 0.1 |d_c^{\mathrm{full}} - d_c^{\mathrm{simp}}|\right).$$ This inequality yields two key properties: 1. **Exact zero difference when both color differences saturate ($d_c^{\mathrm{full}} \ge 5$ and $d_c^{\mathrm{simp}} \ge 5$):** Because the normalized color error $e_c^a = \min(d_c^a / 5, 1)$ clamps to $1.0$ at $d_c^a = 5$, whenever a component reaches or exceeds the threshold of $5$ under both formulas, both normalized color errors saturate to $1.0$ and **$B_{\mathrm{material}}^{\mathrm{full}}(a) - B_{\mathrm{material}}^{\mathrm{simp}}(a) \equiv 0$**. 2. **$0.1\times$ upper bound in the unsaturated regime (at least one $d_c^a < 5$):** A nonzero material-score difference can arise only when at least one formula yields $d_c^a < 5$. In this regime, the per-part material-score discrepancy is **bounded above by $0.1$ times the color-distance discrepancy $|d_c^{\mathrm{full}} - d_c^{\mathrm{simp}}|$ and by $0.5$** ($L = \frac{0.5}{5} = 0.1$), while actual score differences and rankings across benchmark configurations are established by the 31-configuration re-evaluation below.
Reviewer_cvtyOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary
  • The paper introduces PartGround-Bench, benchmark of 1195 manually anotated 3D objects, for evalauting VLM-based 3D reconstruction.
  • The reconstruction task requires VLM to generate Blender Python code from reference images and textual or visual part guidance, producing 3D objects w/ explicitly named components.
  • The proposed multi-agent framework uses a coordinator to plan and assign components to part-building agents, which generate and assemble the geometry. A VLM reviewer then inspects the rendered reconstructions against reference images and suggests a targeted correction (there provide both single-VLM call and multi-call settings with iterative refinements).
  • The benchmark evaluates overall geometry, individual part fidelity, materials and spatial relationships using metrics such as IoU, Part-F1, layout consistency, penetration, contact accuracy.
  • They show that overall global; reconstruction scores can hide significant differences in part quality. Higher reasoning effort improves geometry, while the multi-agent workflow provides gains.
Critical strengths
  • They addresses an important limitation of existing 3D reconstruction benchmarks, where good overall geometry scores can hide errors in individual parts and their assembly.
  • Sharing 1195 manually annotated objects across 12 categories providing a reasonably diverse evalaution set for future VLM evaluations.
  • Fixed part correspondences and shared global alignment is useful, as it evaluates both part geometry, placement etc without independently aligning each component.
  • The evaluation is fairly comprehensive, covering overall geometry, part fidelity, materials, spatial relationships ect.
  • Dense evaluations. Present both single-call VLM results, iterative agentic framework and muklti-agent framework
Critical weaknesses
  • Novelty concern: Works such as P3D-Bench and Hi3DEval already evaluate part-level quality. A direct comparison would better establish the advantages of the proposed evaluation.
  • Part-F1 relies on a fixed IoU threshold of 0.5, which may be sensitive to small placement errors and thin components. Probablly, Human evaluation would help.
  • The coordinator, part builders, and independent reviewer follow ideas similar to Procedura (mentioned in Related work)), a multi-agent framework. The paper does not clearly establish what is new in the proposed framework.
  • The multi-agent approach uses substantially more tokens than the baseline. It is unclear whether the gains come from the proposed coordination strategy or simply additional computation.
Questions for authors

Please address weaknesses mentioned.,

10 comments

Thank you for your thoughtful review and positive assessment (Weak Acceptance), especially for recognizing how our work addresses the limitation of global geometry metrics masking part- and assembly-level errors, our release of 1,195 manually verified multi-part 3D assets across 12 categories, and the methodological value of fixed semantic correspondence under a single shared rigid alignment across geometry, material, and spatial relations. Following your suggestions, we have conducted additional analyses—including multi-threshold sensitivity, a blinded Arena human evaluation, a direct mechanism comparison with Hi3DEval, P3D-Bench, and Procedura, and approximately token-matched controls with component ablations—and updated the revised PDF (highlighted in blue). Below we respond to each concern point by point.

[W1] Direct Comparison with P3D-Bench and Hi3DEval. [Part 1/2]

Thank you for pointing this out. Our initial discussion of Hi3DEval and P3D-Bench in Sections 1 and 2.2 was indeed brief and did not contrast their evaluation mechanisms in sufficient detail. Although both Hi3DEval and P3D-Bench introduce part-aware evaluation, they address fundamentally different questions from PartGround-Bench across evaluation paradigm (reference-free perceptual scoring vs. reference-anchored geometric auditing), part correspondence (post-hoc model segmentation vs. fixed semantic-name contracts), and spatial alignment (independent per-part pose realignment vs. a single globally shared rigid transform TT). PartGround-Bench evaluates whether each specified semantic component is synthesized with accurate 3D shape, PBR material attributes, and inter-part relations at its target assembled position within the whole object—using deterministic white-box geometry without trained evaluators or external splitting models.

First, compared with Hi3DEval's reference-free black-box neural perceptual scoring, PartGround-Bench provides reference-anchored, deterministic white-box diagnostics. Hi3DEval operates on unsegmented generated meshes by partitioning surfaces into class-agnostic clusters via an unsupervised feature field (PartField), predicting perceptual "geometric plausibility" and "detail" scores with a 5M-parameter 3D attention network per scoring dimension, and regressing overall and material scores from orbital videos with a 37M-parameter video network per scoring dimension. While aligned with human perceptual ratings, this reference-free neural regression depends on unsupervised mesh partitioning and cannot isolate per-part dimensional errors, PBR parameter deviations, or geometric interpenetration (ExPen). In contrast, PartGround-Bench requires models to output explicitly named Blender meshes under a fixed semantic specification and compares them deterministically against human-verified 3D reference scenes across in-situ part IoU, PBR attributes (base color, roughness, metallicity), and inter-part spatial relations.

Second, compared with P3D-Bench's post-hoc LLM code splitting, deduplication, and independent per-part pose realignment, PartGround-Bench preserves in-situ assembly placement errors through fixed semantic correspondence and a single shared whole-object transform TT. On P3D-Bench's multi-part subset (203 CAD assemblies), evaluated models output a monolithic program that an external MLLM (Claude Opus 4.6) rewrites post hoc into sub-part scripts (discarding samples that fail a decomposition-fidelity gate), collapses repeated parts (e.g., four identical chair legs) via rotation- and translation-invariant geometric fingerprints, and scores matched candidate pairs (PartFS and PartMatchF1) after independently translating each predicted part's centroid to the ground-truth part centroid and searching over 24 cubic rotations. This per-part realignment isolates local shape recovery by erasing translation offsets and 90∘90^\circ orientation errors, allowing a correctly shaped wheel placed a meter away from the car body to still score high on part metrics. PartGround-Bench instead establishes one-to-one correspondence directly from fixed target part names (requiring neither an external splitting LLM nor a fidelity gate) and estimates a single whole-object rigid transform TT applied identically to all parts (Sections 3.2–3.3). Consequently, relative shape errors, placement offsets, proportion mismatches, and orientation errors are directly captured by in-situ SQ and Part-F1, complemented by pairwise relative layout (ΔAUC\Delta\mathrm{AUC}), reference-calibrated excess penetration (ExPen), and surface contact consistency (ContactF-AUC) across 1,195 assets.

[W1] Direct Comparison with P3D-Bench and Hi3DEval. [part 2/2]

We summarize these differences below and have expanded the final paragraph of Section 2.2 (Evaluation of 3D Generation and Modeling) in the revised PDF (highlighted in blue):

DimensionHi3DEval (NeurIPS 2025)P3D-Bench (2026)PartGround-Bench (Ours)
Paradigm & ScaleReference-free perceptual regression; 15,300 single-mesh generationsReference CAD programs; 203 multi-part assemblies (100 evaluated)Reference-anchored 3D white-box audit; 1,195 human-verified multi-part assets
Part Acquisition & CorrespondenceUnsupervised PartField surface clustering (no semantic names)Post-hoc Claude Opus 4.6 code splitting + fidelity gate + fingerprint deduplication + Hungarian matchingDirect named-mesh output; exact one-to-one correspondence by fixed semantic part names
Alignment for Part ScoringNone (neural feature extraction and score regression)Independent per-part centroid translation + 24-rotation search (isolates local shape)Single whole-object rigid transform TT shared across all parts
Retains In-Situ Placement Error in Part Scores?Implicitly mixed into black-box plausibilityNo (translation and 90∘90^\circ rotation errors are removed during per-part alignment)Yes (in-situ SQ and Part-F1 jointly test part geometry and assembled placement)
Material & Inter-Part Relation MetricsVideo-regressed material score; no explicit inter-part relation metricsWhole-object geometry + MLLM visual judge (J-Geo); no material or inter-part relation metricsDeterministic white-box metrics: matched-part PBR + layout ΔAUC\Delta\mathrm{AUC} + ExPen + ContactF-AUC

[W2] [email protected] Threshold Sensitivity and Blinded Arena Human Evaluation [part 1/3]

To prevent a single hard threshold from overly penalizing thin components or slight placement offsets, our evaluation protocol already combines 1-voxel surface dilation with threshold-free continuous reporting (SQ), and we have now conducted a 4-threshold sensitivity analysis across τ∈{0.10,0.25,0.50,0.75}\tau \in \{0.10, 0.25, 0.50, 0.75\} alongside a blinded Arena human evaluation, presented in full below (and added to Appendix B.3, Table 5 and Figure 5 of the revised PDF).

First, at the geometric representation level, we apply 1-voxel surface dilation on the D/64D/64 surface occupancy grid, where DD is the reference object's axis-aligned bounding-box diagonal in metres (Appendix A.1), to mitigate boundary quantization sensitivity on thin stems, slender legs, and panels; in Tables 1 and 3, we always pair [email protected] with threshold-free continuous SQ (the mean in-situ IoU across matched parts). For instance, under Visual Part Guidance, while Kimi-K3 (max) and Claude-Fable-5 (high) have nearly identical whole-object IoU (0.370 vs. 0.372), continuous SQ (0.187 vs. 0.207) and [email protected] (0.149 vs. 0.168) agree completely on their relative part-level advantage.

Second, to examine sensitivity across looser and stricter thresholds, we report Part-F1 across τ∈{0.10,0.25,0.50,0.75}\tau \in \{0.10, 0.25, 0.50, 0.75\} alongside their four-threshold mean mPart-F1 across all 28 single-call and agent configurations below (also added as Table 5 in Appendix B.3 of the revised PDF):

ConfigurationSettingExec. (%) ↑IoU ↑SQ ↑[email protected] ↑[email protected] ↑[email protected] ↑[email protected] ↑mPart-F1 ↑
Single-call VLM: full benchmark (1,195 objects)
GPT-5.5 (low)Textual95.80.2370.1240.1610.0970.0600.0060.081
GPT-5.5 (low)Visual94.90.2440.1420.1730.1320.0700.0120.097
GPT-5.5 (medium)Textual96.90.2860.1620.1560.1140.0870.0060.091
GPT-5.5 (medium)Visual96.50.2830.1710.1860.1420.0960.0110.109
GPT-5.5 (high)Textual95.40.2940.1740.2030.1630.0990.0100.119
GPT-5.5 (high)Visual97.60.3010.1880.2110.1780.0980.0170.126
GPT-5.5 (xhigh)Textual95.80.3040.1920.2410.1740.1130.0150.136
GPT-5.5 (xhigh)Visual96.40.3030.1990.2600.1810.1120.0200.143
GPT-5.6-Sol (low)Textual95.30.2770.1550.1890.1530.0840.0060.108
GPT-5.6-Sol (low)Visual95.40.2790.1640.2080.1590.0840.0080.115
GPT-5.6-Sol (medium)Textual97.80.2950.1820.1830.1480.1050.0060.111
GPT-5.6-Sol (medium)Visual94.40.2930.1810.2330.1660.1100.0140.131
GPT-5.6-Sol (high)Textual94.20.3120.1890.2220.1550.1200.0140.128
GPT-5.6-Sol (high)Visual94.50.3030.1900.2570.1760.1130.0190.141
GPT-5.6-Sol (xhigh)Textual94.40.3420.2270.2440.1770.1690.0170.152
GPT-5.6-Sol (xhigh)Visual94.80.3290.2150.2690.1890.1540.0220.159
GPT-5.6-Sol (max)Textual94.70.3440.2450.3260.2370.1780.0600.200
GPT-5.6-Sol (max)Visual95.80.3450.3070.3870.2930.1830.0760.235
Agent: subset of 185 objects
Kimi-K2.6 (thinking)Textual100.00.2110.0550.1550.0650.0210.0000.060
Kimi-K2.6 (thinking)Visual100.00.2320.0690.1950.1000.0340.0110.085
Kimi-K3 (max)Textual100.00.3390.1360.3310.2660.0900.0440.183
Kimi-K3 (max)Visual100.00.3700.1870.4170.2920.1490.0570.229
Claude-Fable-5 (high)Textual100.00.3520.1480.3720.2870.1000.0550.204
Claude-Fable-5 (high)Visual100.00.3720.2070.4540.3280.1680.0610.253
GPT-5.6-Sol (xhigh)Textual100.00.2550.0700.1960.1040.0340.0110.086
GPT-5.6-Sol (xhigh)Visual100.00.2680.0840.2260.1330.0430.0120.104
GPT-5.6-Sol (max)Textual100.00.3460.1360.3120.2320.0950.0460.171
GPT-5.6-Sol (max)Visual100.00.3490.1670.3700.2550.1340.0550.204

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.