- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces P3D-Bench, a benchmark for evaluating multimodal large language models on code-based parametric 3D generation. The benchmark covers Text-to-3D, Image-to-3D, and Assembly-3D tasks across multiple program representations. It combines executable validity, geometric fidelity, topology, MLLM-based judgments, and part-level matching. The authors construct a dataset from existing CAD resources and evaluate general-purpose and domain-specific models on 100 cases per task. The results suggest that current models achieve stronger semantic alignment than geometric accuracy and continue to struggle with multi-part reconstruction.
Critical strengths
-
The benchmark provides useful coverage of code-based parametric 3D generation. By combining text, image, and assembly tasks with several program representations, P3D-Bench offers a broader evaluation setting than benchmarks restricted to individual modalities or output formats. Its explicit part-level geometric matching is also a useful addition to existing visual and assembly-oriented evaluations.
-
The evaluation is systematic and provides informative diagnostic results. The paper considers geometric, topological, semantic, and part-level properties, rather than relying on rendering quality alone. The extensive model comparisons, per-format results, qualitative examples, and supplementary analyses help characterize current models' limitations. The authors also provide implementation details and preliminary human validation of the MLLM-based evaluator.
Critical weaknesses
-
The part-level evaluation does not fully establish the claimed assessment of assembly structure. PartMatchF1 operates on geometrically deduplicated parts, and individual part comparisons allow independent rotation and centroid alignment. Consequently, the metric primarily measures matching between distinct part geometries rather than the correctness of all part instances and their spatial relationships. Furthermore, predicted parts are obtained through an additional MLLM-based decomposition step rather than directly evaluated from the original generated program. Although the overall geometry metrics partially capture assembly layout, the current evidence does not fully establish whether the proposed Part metrics reliably assess the recovery of assembly structure.
-
The reliability of the MLLM-based evaluation requires stronger validation. The Judge metrics rely on a single MLLM, Gemini 3.8 Flash, while human validation covers only ten cases per task. Although the reported correlations are encouraging, the limited validation does not fully establish the reliability of individual scoring dimensions or the stability of model rankings.
Questions for authors
-
I am not sure how well the Part metrics capture repeated parts and their spatial relationships after deduplication and independent alignment. Would the results change if part counts and poses were considered directly? Also, how much do the scores depend on the decomposition model and matching threshold?
-
The human evaluation results look promising, but ten cases per task seem rather limited. I wonder whether the individual Judge metrics agree as well with human ratings. Would using another MLLM as the judge lead to similar model rankings?
-
I would also like to see how the models perform on assemblies with different levels of complexity. For example, does performance consistently decrease as the number of parts increases? This could help explain whether the main difficulty comes from assembling multiple parts or reconstructing their geometry.
-
I noticed that IoU is excluded from the Geo score when a valid mesh has open edges, while Topo is not included in the overall Score. Could this make the comparison across output formats less fair, especially for Three.js? Would the model rankings change if such outputs received an explicit penalty?
2 comments
Response to Reviewer F5cq (2/2)
Q3: Does performance decrease as assembly complexity and the number of parts increase?
As complexity increases from 3 to 5–8, Part falls from 44.76 to 35.19 and SpatialF1 from 22.84 to 14.91. Using the revised metrics defined in G4, we report the full breakdown below:
| Grouping | Group | Revised Part | SpatialF1 |
|---|---|---|---|
| Overall complexity | 3 | 44.76 | 22.84 |
| Overall complexity | 4 | 40.70 | 16.73 |
| Overall complexity | 5–8 | 35.19 | 14.91 |
| Distinct parts | 2–5 | 50.43 | 29.06 |
| Distinct parts | 6–10 | 36.71 | 15.21 |
| Distinct parts | 11–20 | 37.72 | 14.51 |
Both scores decrease as assembly complexity increases. They also generally decline as the number of distinct parts increases, with a small rebound in Part for the largest group.
Q4: Would penalizing open meshes or including Topo change model or output-format rankings? We evaluate six combinations: the three IoU rules A, B, and C, each with Topo excluded (suffix 0) or included with equal weight (suffix 1). Across these settings, the maximum generator-rank changes are zero, three, and one positions for Text, Image, and Assembly, and GPT-6 Astra remains first. Format changes are larger: adding Topo moves Three.js from first to fourth in the Gemini Image comparison. See G5 for the full results.
Response to Reviewer F5cq (1/2)
W1: Do the Part metrics fully evaluate assembly structure, including repeated instances and their placement? The original deduplicated metrics measure distinct part geometry. We now add SpatialF1 to retain repeated instances and evaluate their placement after whole-assembly alignment, including penalties for missing, extra, or misplaced instances. The best generator scores 35.55, compared with its geometric PartMatchF1 of 60.75. See G3 and G4 for decomposition validation and the updated metric.
W2 / Q2: Is evaluation with a single MLLM judge and only ten human-evaluated cases reliable, and do the Judge scores agree with human ratings and yield similar model rankings? We expand human validation by randomly selecting 50 predictions for each of Text, Image, and Assembly. With the updated study procedure, Text QA-S human–Gemini answer agreement reaches 91.5%, while human–Gemini Spearman correlations are 0.9321 for Image and 0.9405 for Assembly. For the cross-model score comparison, Pearson correlations between Gemini's scores and the mean score of the three MLLMs are 0.901, 0.902, and 0.917 for Text, Image, and Assembly, respectively. The corresponding generator-rank correlations are 0.879, 1.000, and 0.983. Together, these results support both human agreement and cross-Judge consistency in model rankings. See G1 and G2.
Q1: Would the results change if repeated parts and poses were evaluated directly, or if the decomposer and matching thresholds changed?
SpatialF1 now directly scores repeated instances and their poses. Across decomposers, all-instance F1 ranges from 91.37 to 93.57 on the common 45 predictions passing the fidelity checks. See G3 and G4; the threshold results below show maximum rank changes of one position for Part and two for SpatialF1.
We vary geometric matching and spatial thresholds separately, fixing the other settings and the evaluation cohort. The tables list all combinations. Each cell reports “generator-ranking Spearman relative to the default / maximum rank change”; bold cells indicate the default settings.
Geometric matching: 20 settings. Distance tolerances are 2%, 3%, 4%, and 5% of the GT part's bounding-box diagonal, and matching acceptance thresholds are 0.70, 0.75, 0.80, 0.85, and 0.90. We evaluate every combination. Results refer to the three-component Part score, with its spatial component fixed at the default setting. The default geometric setting is 3% / 0.70.
Spatial matching: 27 settings. Translation tolerances are 10%, 15%, and 20% of the longest side of the GT assembly's bounding box. The symmetric scale-factor limit k is 1.10, 1.15, or 1.20, giving an allowed scale-correction interval of [1/k,k]. Rotation tolerances are 15°, 30°, and 45°. We evaluate all combinations while holding geometric correspondences and fitted transformations fixed. Results refer to SpatialF1; the default setting is 15% / 1.15 / 30°.
Across the 20 geometric settings, the minimum Spearman correlation of the three-component Part ranking with the default is 0.967, with a maximum change of one position. Across the 27 spatial settings, the minimum SpatialF1 rank correlation is 0.917, with a maximum change of two positions. GPT-6 Astra remains first in every setting, and the top three SpatialF1 positions are unchanged, supporting the robustness of the main model comparisons to these specific threshold choices.