acceptodds
Under review as a conference paper at ICLR 2027

P3D-Bench: Benchmarking MLLMs for Parametric 3D Generation and Structural Reasoning

Abstract

Multimodal large language models can write code to produce complex programs as well as use programs to do 3D modeling, which opens up a new avenue for 3D generation powered by their priors, world knowledge and reasoning. Yet existing benchmarks rarely evaluate 3D modeling through code. Such modeling demands more than runnable code: from a text or visual specification, a model must generate a parametric 3D program that is geometrically precise, semantically aligned and assembly-consistent. We introduce P3D-Bench, a benchmark for parametric 3D generation. Unlike a 3D mesh, a parametric 3D program exposes explicit dimensions, construction operations and part relations, revealing whether a model recovers a design's structure, not just its appearance. Under a unified protocol, P3D-Bench covers three task families (Text-to-3D, Image-to-3D and Assembly-3D) and scores each output for executability, geometric fidelity, topology, text-grounded constraints, multiview semantic alignment and part-level structure. We construct P3D-Dataset, comprising 400 text cases, 400 image cases, and 203 annotated assemblies. Our evaluation on 100 cases from each task family yields three key findings. First, multi-part generation is substantially more challenging than single-part modeling, with models struggling to compose individual parts into a coherent structure. Second, models can often recover the global shape and semantic identity of the target object, yet fail to reproduce the precise parametric geometry specified by the input. Third, part-level modeling remains weak on assemblies, where models recover neither the geometry of each part nor the right number of parts. These results position P3D-Bench as a benchmark for evaluating precise parametric geometry and part-level structure in parametric 3D generation.

Primary area:
datasets and benchmarks
Keywords:
Parametric 3D generation, multimodal large language models, 3D program synthesis, structural reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_F5cqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces P3D-Bench, a benchmark for evaluating multimodal large language models on code-based parametric 3D generation. The benchmark covers Text-to-3D, Image-to-3D, and Assembly-3D tasks across multiple program representations. It combines executable validity, geometric fidelity, topology, MLLM-based judgments, and part-level matching. The authors construct a dataset from existing CAD resources and evaluate general-purpose and domain-specific models on 100 cases per task. The results suggest that current models achieve stronger semantic alignment than geometric accuracy and continue to struggle with multi-part reconstruction.

Critical strengths
  1. The benchmark provides useful coverage of code-based parametric 3D generation. By combining text, image, and assembly tasks with several program representations, P3D-Bench offers a broader evaluation setting than benchmarks restricted to individual modalities or output formats. Its explicit part-level geometric matching is also a useful addition to existing visual and assembly-oriented evaluations.

  2. The evaluation is systematic and provides informative diagnostic results. The paper considers geometric, topological, semantic, and part-level properties, rather than relying on rendering quality alone. The extensive model comparisons, per-format results, qualitative examples, and supplementary analyses help characterize current models' limitations. The authors also provide implementation details and preliminary human validation of the MLLM-based evaluator.

Critical weaknesses
  1. The part-level evaluation does not fully establish the claimed assessment of assembly structure. PartMatchF1 operates on geometrically deduplicated parts, and individual part comparisons allow independent rotation and centroid alignment. Consequently, the metric primarily measures matching between distinct part geometries rather than the correctness of all part instances and their spatial relationships. Furthermore, predicted parts are obtained through an additional MLLM-based decomposition step rather than directly evaluated from the original generated program. Although the overall geometry metrics partially capture assembly layout, the current evidence does not fully establish whether the proposed Part metrics reliably assess the recovery of assembly structure.

  2. The reliability of the MLLM-based evaluation requires stronger validation. The Judge metrics rely on a single MLLM, Gemini 3.8 Flash, while human validation covers only ten cases per task. Although the reported correlations are encouraging, the limited validation does not fully establish the reliability of individual scoring dimensions or the stability of model rankings.

Questions for authors
  1. I am not sure how well the Part metrics capture repeated parts and their spatial relationships after deduplication and independent alignment. Would the results change if part counts and poses were considered directly? Also, how much do the scores depend on the decomposition model and matching threshold?

  2. The human evaluation results look promising, but ten cases per task seem rather limited. I wonder whether the individual Judge metrics agree as well with human ratings. Would using another MLLM as the judge lead to similar model rankings?

  3. I would also like to see how the models perform on assemblies with different levels of complexity. For example, does performance consistently decrease as the number of parts increases? This could help explain whether the main difficulty comes from assembling multiple parts or reconstructing their geometry.

  4. I noticed that IoU is excluded from the Geo score when a valid mesh has open edges, while Topo is not included in the overall Score. Could this make the comparison across output formats less fair, especially for Three.js? Would the model rankings change if such outputs received an explicit penalty?

2 comments
AuthorsOpenReview ↗

Response to Reviewer F5cq (1/2)

W1: Do the Part metrics fully evaluate assembly structure, including repeated instances and their placement? The original deduplicated metrics measure distinct part geometry. We now add SpatialF1 to retain repeated instances and evaluate their placement after whole-assembly alignment, including penalties for missing, extra, or misplaced instances. The best generator scores 35.55, compared with its geometric PartMatchF1 of 60.75. See G3 and G4 for decomposition validation and the updated metric.

W2 / Q2: Is evaluation with a single MLLM judge and only ten human-evaluated cases reliable, and do the Judge scores agree with human ratings and yield similar model rankings? We expand human validation by randomly selecting 50 predictions for each of Text, Image, and Assembly. With the updated study procedure, Text QA-S human–Gemini answer agreement reaches 91.5%, while human–Gemini Spearman correlations are 0.9321 for Image and 0.9405 for Assembly. For the cross-model score comparison, Pearson correlations between Gemini's scores and the mean score of the three MLLMs are 0.901, 0.902, and 0.917 for Text, Image, and Assembly, respectively. The corresponding generator-rank correlations are 0.879, 1.000, and 0.983. Together, these results support both human agreement and cross-Judge consistency in model rankings. See G1 and G2.

Q1: Would the results change if repeated parts and poses were evaluated directly, or if the decomposer and matching thresholds changed?

SpatialF1 now directly scores repeated instances and their poses. Across decomposers, all-instance F1 ranges from 91.37 to 93.57 on the common 45 predictions passing the fidelity checks. See G3 and G4; the threshold results below show maximum rank changes of one position for Part and two for SpatialF1.

We vary geometric matching and spatial thresholds separately, fixing the other settings and the evaluation cohort. The tables list all combinations. Each cell reports “generator-ranking Spearman relative to the default / maximum rank change”; bold cells indicate the default settings.

Geometric matching: 20 settings. Distance tolerances are 2%, 3%, 4%, and 5% of the GT part's bounding-box diagonal, and matching acceptance thresholds are 0.70, 0.75, 0.80, 0.85, and 0.90. We evaluate every combination. Results refer to the three-component Part score, with its spatial component fixed at the default setting. The default geometric setting is 3% / 0.70.

Distance tolerance / acceptance threshold0.700.750.800.850.90
2%0.983 / 10.983 / 10.983 / 10.983 / 10.967 / 1
3%1.000 / 00.983 / 11.000 / 01.000 / 00.983 / 1
4%0.967 / 10.967 / 10.983 / 10.983 / 10.983 / 1
5%0.967 / 10.967 / 10.967 / 10.967 / 10.967 / 1

Spatial matching: 27 settings. Translation tolerances are 10%, 15%, and 20% of the longest side of the GT assembly's bounding box. The symmetric scale-factor limit kk is 1.10, 1.15, or 1.20, giving an allowed scale-correction interval of [1/k,k][1/k,k]. Rotation tolerances are 15°, 30°, and 45°. We evaluate all combinations while holding geometric correspondences and fitted transformations fixed. Results refer to SpatialF1; the default setting is 15% / 1.15 / 30°.

Translation toleranceScale-factor limit kRotation 15°Rotation 30°Rotation 45°
10%1.101.000 / 00.983 / 10.983 / 1
10%1.150.933 / 20.950 / 20.933 / 2
10%1.200.950 / 20.950 / 20.946 / 2
15%1.100.983 / 10.983 / 10.983 / 1
15%1.150.983 / 11.000 / 00.983 / 1
15%1.200.983 / 10.983 / 11.000 / 0
20%1.100.983 / 11.000 / 00.983 / 1
20%1.150.983 / 10.983 / 10.967 / 1
20%1.200.917 / 20.950 / 20.983 / 1

Across the 20 geometric settings, the minimum Spearman correlation of the three-component Part ranking with the default is 0.967, with a maximum change of one position. Across the 27 spatial settings, the minimum SpatialF1 rank correlation is 0.917, with a maximum change of two positions. GPT-6 Astra remains first in every setting, and the top three SpatialF1 positions are unchanged, supporting the robustness of the main model comparisons to these specific threshold choices.

AuthorsOpenReview ↗

Response to Reviewer F5cq (2/2)

Q3: Does performance decrease as assembly complexity and the number of parts increase?

As complexity increases from 3 to 5–8, Part falls from 44.76 to 35.19 and SpatialF1 from 22.84 to 14.91. Using the revised metrics defined in G4, we report the full breakdown below:

GroupingGroupRevised PartSpatialF1
Overall complexity344.7622.84
Overall complexity440.7016.73
Overall complexity5–835.1914.91
Distinct parts2–550.4329.06
Distinct parts6–1036.7115.21
Distinct parts11–2037.7214.51

Both scores decrease as assembly complexity increases. They also generally decline as the number of distinct parts increases, with a small rebound in Part for the largest group.

Q4: Would penalizing open meshes or including Topo change model or output-format rankings? We evaluate six combinations: the three IoU rules A, B, and C, each with Topo excluded (suffix 0) or included with equal weight (suffix 1). Across these settings, the maximum generator-rank changes are zero, three, and one positions for Text, Image, and Assembly, and GPT-6 Astra remains first. Format changes are larger: adding Topo moves Three.js from first to fourth in the Gemini Image comparison. See G5 for the full results.

Reviewer_GDBKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

P3D-Bench evaluates code-based 3D generation across Text-to-3D, Image-to-3D, and Assembly-3D tasks, using task-specific combinations of four output formats. It constructs filtered and annotated subsets from Text2CAD and Fusion 360 Gallery and evaluates 100 cases per task. The evaluation combines executable validity, geometry, topology, model-based judgments, and part matching. Results suggest that producing runnable programs is easier than recovering accurate geometry and assembly structure.

Critical strengths
  1. The evaluation goes beyond executability and exposes failures that validity-based metrics would miss: models that produce valid programs almost always still fall well short on geometric accuracy and part structure. This gives the benchmark diagnostic value for future work on parametric 3D generation.

  2. P3D-Bench goes beyond visual judgment by matching predicted and ground-truth parts using part-level F-scores. This provides a direct measure of part recovery that complements whole-shape similarity and helps identify structural errors.

  3. This paper is pretty well-organized and easy to follow.

Critical weaknesses
  1. The main evaluation uses 100 cases per task, and seven models score between 80.1 and 82.4 on Text-to-3D (Sec. 4.1). However, no confidence intervals or analysis of ranking stability are reported for these comparisons, making it difficult to tell whether the small differences reflect consistent performance gaps.

  2. The shared decomposer and fidelity gate help preserve the original predicted geometry, but do not establish whether the original part boundaries are preserved. Parts may be split or merged without changing the overall shape. The reported Part scores, and the conclusion that models fail to recover the correct number of distinct parts, may therefore reflect decomposition errors as well as generation errors.

  3. The reliability of the QA-based Judge needs further support. In the human study, annotators’ answers agree with the Judge’s on only 68.3% of QA-S items for predicted renders, while on ground-truth renders they match the answer key only 80.0% of the time, based on just 10 cases. Since even ground-truth answers are not consistently reproducible by humans, it is unclear whether disagreements stem from ambiguous questions or from Judge errors, which matters when interpreting the reported Text-to-3D scores.

Questions for authors
  1. Do the model and output-format rankings remain stable when all predictions use the same Geo components?

  2. For outputs that explicitly define individual parts, how often does the decomposition MLLM change the original part boundaries?

3 comments
AuthorsOpenReview ↗

Response to Reviewer GDBK

W1: How reliable are the small Text score differences on 100 cases?

We thank the reviewer for pointing out the missing uncertainty estimates. We compute the mean Text score for each generator over the original 100 CAD cases. Using these 100 per-CAD scores, we report the standard two-sided 95% t-confidence interval for the mean (sample standard deviation over the 100 cases, 99 degrees of freedom):

GeneratorOriginal Text score95% CI for the mean
GPT-6 Astra84.73[82.66, 86.79]
Qwen 3.8 Max82.38[80.23, 84.54]
Gemini 3.8 Flash82.10[80.21, 83.99]
Kimi K381.73[79.56, 83.89]
Grok 4.681.40[79.31, 83.48]
Claude Opus 581.13[78.99, 83.26]
GLM 5.380.27[78.28, 82.27]
DeepSeek V4.1 Flash80.13[77.81, 82.45]
Doubao Seed 2.1 Pro79.17[77.08, 81.26]
GLM 5.3 Flash76.35[74.09, 78.61]

Because all generators are evaluated on the same 100 CAD cases, we also compute paired t-confidence intervals for the closest model differences. For each pair, we form the 100 per-CAD score differences and use a two-sided t interval with 99 degrees of freedom:

Comparison (A − B)Mean difference95% CI for the difference
GPT-6 Astra − Qwen 3.8 Max2.34[1.42, 3.26]
Qwen 3.8 Max − Gemini 3.8 Flash0.28[−0.66, 1.22]
GPT-6 Astra − Gemini 3.8 Flash2.62[1.66, 3.59]

The intervals for GPT−Qwen and GPT−Gemini exclude zero, whereas the interval for Qwen−Gemini includes zero. Thus the two former gaps are supported by the paired analysis, while the small Qwen–Gemini gap is not distinguishable from zero at this level.

We agree that some models have similar Text scores. This task is relatively simple for the evaluated models, which helps explain the small score differences. The Text generator rankings from Gemini and the mean score of the three MLLMs have a Spearman correlation of 0.879, and Text QA-S human agreement reaches 91.5%. These results support broadly consistent Text evaluation trends; see G1 and G2.

W2 / Q2: Does decomposition preserve part boundaries in the original generated programs? The direct part comparison on 50 predictions gives deduplicated F1 scores of 100.00/100.00/98.00 for Gemini/GPT/DeepSeek, checking the extracted parts rather than only the whole shape. See G3 for the source-program validation and cross-decomposer results.

W3: Why were QA-S agreement rates low on both GT and predicted renders? We expand the Text human evaluation to 50 randomly sampled predictions and 50 GT renders. Human–reference answer agreement on GT reaches 98.0%, compared with 80.0% originally, while human–Gemini answer agreement on predictions reaches 91.5%, compared with 68.3%. These results support the consistency of the reference answers and Gemini’s judgments, respectively. See G2 for the evaluation procedure and details.

Q1: Do model and output-format rankings remain stable when all predictions use the same Geo components? Using the same three surface components for every prediction preserves all Text and revised-Assembly generator ranks and swaps only one adjacent Image pair. Format rankings can change: in the Gemini four-format Image experiment, Three.js moves from first to third. See G5 for the complete model and format comparisons.

Thanks for authors' feedback, which has addressed most of my concerns and questions. I have no further questions.

Thank you for your careful review and valuable feedback. We are pleased that our responses have addressed your concerns and questions. We sincerely appreciate the time and effort you devoted to evaluating our work.

Reviewer_G6jWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes P3D-Bench, a benchmark evaluating MLLMs on their capability in parametric 3D generation. The benchmark emphasizes on explicit geometry precision and assemply part generation, and proposes evaluation metrics covering validity, geometry, topology, LLM judge and part-level accuracy.

Critical strengths
  1. the benchmark has significant practical potentials. Generating geometrically precise 3D parts is an important aspect that carries values in industry and also measures MLLMs' advancements in 3D understanding and structural reasoning.

  2. The benchmark covers multiple output format and reports evaluations of several most recent models, with a suite of evaluation metrics from validity to precision. It is a comprehensive experiment.

Critical weaknesses
  1. One of the major concern is the Part metric. As described in section 3.3, the Assembly-3D Part Metric relies on a "fixed decomposition MLLM" to split assembly into part programs. How reliable and acurate is this part-decomposition? Also the evaluation invovles a deduplication of parts and only evaluate distinct part. But it is common for an assembly to actually have multiple duplicate parts but assembled to different locations. How do the evaluation measure the "structural reasoning" capability if this is overlooked?

  2. Implicit bias in LLM-as-judge. Gemini 3.8 Flash is chosen as the judge without enough justification and its agreement with human on QA-S is only 68.3% in table 3b. And human raters only accessed 10 cases for each task.

Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer G6jW

We thank the reviewer for the concerns about decomposition reliability, repeated parts, and judge validation.

W1: How reliable is part decomposition, and how are repeated parts and their positions evaluated? Direct comparison with parts exported from the original programs gives deduplicated F1 scores of 100.00, 100.00, and 98.00 for Gemini, GPT, and DeepSeek. The new SpatialF1 evaluates every part instance and its placement, including repetitions; the best generator scores 35.55. See G3 and G4 for the validation and updated results.

W2: Why use Gemini as the judge, and is the human validation sufficient? We expand the human evaluation by randomly selecting 50 predictions for each of Text, Image, and Assembly. With the updated study procedure, Text QA-S human–Gemini answer agreement reaches 91.5%. The Image and Assembly samples yield human–Gemini Spearman correlations of 0.9321 and 0.9405, respectively. Across the three tasks, Gemini's scores correlate with the mean score of the three MLLMs at 0.901–0.917. See G1 and G2.

Thank you for the answers, which have addressed my concerns. I therefore would like to raise my rating.

Thank you very much for your follow-up and for raising your rating. We sincerely appreciate your careful evaluation and constructive comments, which have helped us improve the manuscript.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response (1/4)

We thank the reviewers for their constructive feedback on evaluation reliability, part-level evaluation, and the robustness of our findings. We have added comparisons across judges, human validation, decomposition validation, instance-level spatial evaluation, and sensitivity analyses for IoU and Topo. We first address the shared concerns below, followed by reviewer-specific responses. We will incorporate the cross-judge comparison results, human-evaluation and decomposition validation results, SpatialF1, the revised Part definition, and the corresponding updated benchmark results into the revised PDF shortly.

G1. Why use Gemini as the judge, and would other MLLMs give similar scores and rankings?

We compare Gemini 3.8 Flash, GPT-6.1 Sol, and DeepSeek V4.1 Flash on the same generated outputs across the full 100-case evaluation set of each task: Text-to-3D, Image-to-3D, and Assembly-3D. We retain each task's validity and missing-data rules. The comparison examines both agreement in overall Judge scores and stability of generator rankings.

For each evaluation unit, we compute the equally weighted mean of the overall Judge scores from Gemini, GPT, and DeepSeek. We refer to this as the mean score of the three MLLMs:

Javg=JGemini+JGPT+JDeepSeek3.J_{\mathrm{avg}}=\frac{J_{\mathrm{Gemini}}+J_{\mathrm{GPT}}+J_{\mathrm{DeepSeek}}}{3}.

Text follows the original paper's overall Judge definition; Image and Assembly use their final Visual Judge scores. These results do not incorporate Geo, Part, or the overall task score.

TaskPearson: Gemini vs. mean score of the three MLLMs
Text-to-3D0.901
Image-to-3D0.902
Assembly-3D0.917

We also report Pearson correlations between Gemini and the mean score of the three MLLMs for each scoring component:

TaskScoring componentPearson: Gemini vs. mean score of the three MLLMs
TextDescriptive QA-S0.921
TextDescriptive J-Sem0.875
TextParametric QA-S0.881
TextParametric QA-P0.897
ImageJ-Geo0.855
ImageJ-Aes0.878
ImageJ-Sem0.858
AssemblyJ-Geo0.872
AssemblyJ-Aes0.889
AssemblyJ-Sem0.893

We further compare generator rankings obtained from Gemini and the mean score of the three MLLMs:

GeneratorText: GeminiText: Mean score of the three MLLMsImage: GeminiImage: Mean score of the three MLLMsAssembly: GeminiAssembly: Mean score of the three MLLMs
GPT-6 Astra111111
Gemini 3.8 Flash44→22222
Claude Opus 5553333
Grok 4.666→45544
Kimi K333→64455
Qwen 3.8 Max22→36677→8
GLM 5.3 Flash10107766
DeepSeek V4.1 Flash88→78888→7
Doubao Seed 2.1 Pro999999
GLM 5.377→8————

In the columns based on the mean score of the three MLLMs, bold entries show Gemini rank→rank from the mean score of the three MLLMs when the rank changes; unchanged ranks remain single numbers. “—” indicates that no ranking is reported for that model and task.

The Spearman correlations between rankings based on Gemini's scores and the mean score of the three MLLMs are 0.879, 1.000, and 0.983 for Text, Image, and Assembly, respectively. GPT-6 Astra remains first in all three tasks. All Image ranks agree; Assembly has one adjacent swap, while Text changes are concentrated among intermediate generators. The score and ranking comparisons together show substantial consistency between Gemini and the mean score of the three MLLMs.

AuthorsOpenReview ↗

General Response (2/4)

G2. Is human validation on only ten cases sufficient, and why is Text QA-S agreement low?

We thank the reviewers for raising concerns about the small human-study sample and the low Text QA-S agreement. When revisiting the original Text study, we found that the input text was hidden by default, the instructions were unclear, and the eight GT and prediction views were shown together. This made it harder to find the relevant features and choose an answer. We now display the full input text directly and evaluate GT and predictions separately. The instructions explain that the text provides context, but answers must reflect the geometry visible in the current renders. This procedure aims to reduce confusion caused by missing context or by treating requested features as if they had actually been generated.

We increase the human-evaluation sample size by randomly selecting 50 predictions for each of Text, Image, and Assembly. For Text QA-S, we measure answer agreement between humans and Gemini on the predicted renders. We also measure agreement between human answers and the reference answer key on 50 GT renders. For Image and Assembly, we compute the Spearman correlation between Gemini's raw Visual scores and mean human scores.

Task and evidenceSamplesComparisonAgreement measureResult
Text QA-S: predicted renders50Humans vs. GeminiAnswer agreement91.5%
Text QA-S: GT renders50Humans vs. reference answersAnswer agreement98.0%
Image Visual Judge50Humans vs. GeminiSpearman0.9321
Assembly Visual Judge50Humans vs. GeminiSpearman0.9405

With the updated procedure and expanded sample, Text QA-S human–judge agreement on predicted renders increases from 68.3% to 91.5%, while human agreement with the reference answers on GT renders increases from 80.0% to 98.0%. People reproduce the GT answer key more consistently, and Gemini's answers on generated outputs agree more closely with human answers. Image and Assembly also show high correlations with human ratings, at 0.9321 and 0.9405, respectively.

G3. How reliable is MLLM-based part decomposition, and does it preserve the original part boundaries?

We agree that preserving the overall shape does not guarantee correct part boundaries. We therefore compare the extracted parts with parts identified in the original generated programs, and compare different decomposers on randomly sampled predictions.

We found that the previous decomposition procedure was not sufficiently accurate for some predictions. On 50 predictions whose individual parts can be identified in the original code, the original Gemini, GPT, and DeepSeek decomposers matched the reference instance count in 45/50, 46/50, and 42/50 cases, respectively. We improved the prompt by adding the part names and requested counts from the original generation input. This gives the model clearer guidance about the parts to separate. It must preserve the generated geometry and must not add parts to meet the requested counts.

First, we export the identified parts directly from the original code for these 50 predictions and compare them with the decomposer's outputs:

DecomposerInstance-count agreementDeduplicated part F1All-instance F1
Gemini 3.8 Flash50/50100.00100.00
GPT-6.1 Sol50/50100.00100.00
DeepSeek V4.1 Flash49/5098.0098.00

This comparison checks the number and geometry of the extracted parts against the original program. All three decomposers show high agreement under the benchmark's geometric tolerance and matching threshold.

Second, we randomly sample 50 prediction programs. Of these, 45 satisfy successful execution, complete part export, and the whole-shape fidelity conditions for all three decomposers. Results on this identical subset are:

Decomposer pairDeduplicated part F1All-instance F1
Gemini / GPT94.0593.54
Gemini / DeepSeek91.3491.37
GPT / DeepSeek93.5293.57

These checks support the reliability of the improved decomposition procedure. The extracted parts closely match those from the original programs, and different MLLMs produce highly similar decompositions on randomly sampled predictions.

AuthorsOpenReview ↗

General Response (3/4)

G4. How are repeated parts and their positions evaluated after deduplication?

We agree that the original deduplicated Part metrics primarily assess the recovery of distinct part geometries and do not fully capture repeated-instance counts and placement. We therefore introduce SpatialF1, retaining all instances, and incorporate it into the Part score.

The evaluation proceeds as follows:

  1. Match deduplicated part types using the original Part procedure. We first align the predicted assembly with the GT assembly, then geometrically match their deduplicated part types to establish type-level correspondences.

  2. Match all instances within each type. We retain every instance, including repetitions, and match prediction and GT instances within each corresponding part group using their XYZ positions. This yields instance-level correspondences without deduplication.

  3. Evaluate placement errors. For each matched instance pair, we measure the translation, scale, and rotation corrections needed to reach the reference placement.

An instance pair is successful only if all three errors meet their thresholds. The defaults are a translation error no greater than 15% of the longest side of the GT assembly's bounding box, a scale correction factor in [1/1.15, 1.15], and a rotation error no greater than 30°, accounting for geometric symmetry. SpatialF1 is defined as:

SpatialF1=2MNpred+NGT,\mathrm{SpatialF1}=\frac{2M}{N_{\mathrm{pred}}+N_{\mathrm{GT}}},

where MM is the number of successful instance matches. All predicted and GT instances remain in the denominator, so missing parts, extra instances, and incorrect placements reduce the score. For reporting and Part-score aggregation, SpatialF1 is multiplied by 100 so that all three Part components use the same 0–100 scale.

The revised Part score is:

=\frac{\mathrm{PartFS}+\mathrm{PartMatchF1}+\mathrm{SpatialF1}}{3}.$$ We recompute the results on the original Assembly100 evaluation set, retaining the original Geo and Judge components and the task-score aggregation structure: | Generator | Revised Part | SpatialF1 | Revised Assembly score | Original → revised rank | |:---:|:---:|:---:|:---:|:---:| | GPT-6 Astra | 57.26 | 35.55 | 61.78 | 1 → 1 | | Gemini 3.8 Flash | 43.71 | 22.78 | 50.90 | 3 → 2 | | Claude Opus 5 | 44.00 | 19.10 | 50.78 | 2 → 3 | | Grok 4.6 | 38.81 | 15.72 | 47.78 | 5 → 4 | | Kimi K3 | 40.57 | 16.57 | 47.21 | 4 → 5 | | Qwen 3.8 Max | 40.60 | 16.67 | 46.72 | 6 → 6 | | GLM 5.3 Flash | 38.74 | 15.76 | 46.46 | 7 → 7 | | DeepSeek V4.1 Flash | 37.13 | 15.91 | 44.68 | 8 → 8 | | Doubao Seed 2.1 Pro | 22.70 | 6.87 | 27.54 | 9 → 9 | The leading generator remains unchanged, while two adjacent pairs exchange positions. Gemini and Claude differ by only approximately 0.12 points in the revised overall score, which we do not interpret as an established superiority of one over the other. Even under these relatively permissive spatial thresholds, the best SpatialF1 is only **35.55**, compared with a deduplicated PartMatchF1 of **60.75** for the same model. Recovering part geometry therefore does not imply correctly recovering all instances and their placement. ### G5. Do the IoU and Topo scoring choices affect fairness across models and formats, especially Three.js? An open mesh does not reliably define an enclosed volume, so volumetric IoU may be unavailable or unreliable. The original protocol retains the available surface metrics in this situation. Topo directly measures mesh-integrity properties, including open boundaries, inverted normals, and non-manifold edges, and is reported separately in the original paper. Using the same 100-case evaluation sets and saved outputs, we hold the Gemini Judge and other score components fixed and jointly examine three IoU rules and whether Topo is included as an equally weighted component. Text and Image use the original paper's protocol; Assembly fixes the revised three-component Part score. | Rule | IoU treatment | |:---:|:---:| | A | Original: average four components when IoU is available and three surface components otherwise | | B | Remove IoU uniformly for all predictions | | C | Set unavailable IoU to zero; otherwise retain A | Suffix 0 excludes Topo and suffix 1 includes it with equal weight, giving six settings: A0, A1, B0, B1, C0, and C1. When included, Topo has weights of 1/4, 1/3, and 1/4 for Text, Image, and Assembly, respectively.
AuthorsOpenReview ↗

General Response (4/4)

1. Generator rankings

All comparisons below use the corresponding task's original rule A0 as the baseline. Each cell reports “rank Spearman / maximum rank change”:

SettingTextImageAssembly
A01.000 / 01.000 / 01.000 / 0
B01.000 / 00.983 / 11.000 / 0
C01.000 / 00.983 / 11.000 / 0
A11.000 / 00.900 / 20.983 / 1
B11.000 / 00.900 / 20.983 / 1
C11.000 / 00.833 / 30.967 / 1

The generator ranks under each setting are listed below:

GeneratorAssembly-3DText-to-3DImage-to-3D
A0B0C0A1B1C1A0B0C0A1B1C1A0B0C0A1B1C1
GPT-6 Astra111111111111111111
Gemini 3.8 Flash2222→32→32→33333332222→32→32→4
Claude Opus 53333→23→23→26666663333→23→23→2
Grok 4.644444455555555→65→65→75→75→7
Kimi K3555555→64444444444→54→54→5
Qwen 3.8 Max666666→522222266→56→56→46→46→3
GLM 5.3 Flash7777771010101010107777→67→67→6
DeepSeek V4.1 Flash888888888888888888
Doubao Seed 2.1 Pro999999999999999999
GLM 5.3——————777777——————

Bold entries show A0 rank→new rank for the same task; unchanged ranks remain single numbers. “—” indicates that no ranking is reported for that model and task.

Changing only the IoU rule preserves the Text and Assembly rankings and swaps only the adjacent Qwen/Grok pair for Image. When Topo is jointly included, Text remains unchanged, the maximum rank change is three positions for Image and one for Assembly, and GPT-6 Astra remains first in every setting.

Image rankings change more because the models have different Topo scores. Including Topo can move models with poorer mesh quality down in the ranking. Geo, Judge, and Part mainly measure how closely the generated object matches the target, while Topo checks the quality of the mesh itself. We therefore report Topo separately and use the comparison above to show its effect on the overall score.

2. Output formats and Three.js

The table below reports overall scores for the four formats produced by Gemini as the generator. A0/B0/C0 compare IoU rules; A1 shows the effect of including Topo under the original IoU rule:

TaskFormatA0B0C0A1
TextOpenSCAD83.7284.0983.7287.79
TextJSON80.4881.0280.2485.02
TextThree.js78.5278.8677.7381.29
TextCadQuery72.1972.6471.9976.56
ImageThree.js53.9554.2148.1859.62
ImageOpenSCAD53.8855.0353.8869.24
ImageCadQuery53.2954.4452.8867.57
ImageJSON41.0942.1741.0960.71
AssemblyThree.js51.7352.0547.9257.36
AssemblyOpenSCAD51.6352.7251.6363.72
AssemblyCadQuery50.1851.2049.7961.33
AssemblyJSON39.8040.9339.5954.54

For Image, Three.js moves from first under A0 to third under B0/C0 and fourth under A1 when Topo is included. For Assembly, it moves from first to second under B0 and third under C0/A1.

This behavior is consistent with Figure 6 of the original paper: Three.js constructs triangulated surfaces, whose generated meshes are more prone to open boundaries and have lower Topo scores than the other formats. Its relative rank therefore decreases as expected when prediction-open cases are penalized or mesh integrity is included in the overall score. The broad model-ranking trends remain similar when each evaluation rule is applied consistently. Differences between formats reflect the properties and strengths of the formats themselves.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.