acceptodds
Withdrawn from ICLR 2027

MUSE: Benchmarking Assembly Structure and Interface Geometry in Multi-Component Text-to-CAD Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) have recently advanced text-driven 3D generation, yet generating CAD models that satisfy practical design requirements remains challenging. Existing benchmarks focus primarily on single-part CAD models and geometric similarity, providing limited assessment of assembly structure and interface geometry. To address this gap, we introduce \model, a Text-to-CAD benchmark for generating multi-component, editable boundary representation (B-Rep) assemblies. \model pairs practical design instances with structured Design Specifications and evaluates generated outputs across four dimensions: code executability, geometric validity, structural and interface fidelity, and functional consistency. We further introduce Structure and Interface Evaluation (SIE), which combines reference Physical Assembly Graphs with direct B-Rep analysis to assess component correspondence, connectivity, interface types, and fit clearances. Experiments on proprietary and open-weight LLMs show that high execution success does not necessarily translate into correct assembly structure or interface geometry. Together, \model provides a benchmark and evaluation framework for studying multi-component Text-to-CAD generation beyond shape similarity. Our dataset and code are available at https://anonymous.4open.science/status/muse_-8C3B.

Primary area:
datasets and benchmarks
Keywords:
Text-to-CAD, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_B6S6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MUSE, a benchmark for CAD generation from structured design specifications, with a focus on assembly structure and interface geometry in multi-component models. The dataset contains 106 design instances constructed through expert seed modeling, LLM-assisted augmentation, and expert review. Each input includes a natural-language design description and a physical assembly graph; the model generates a CadQuery program that exports a STEP file. The core method, Structure and Interface Evaluation (SIE), combines reference-component matching with direct B-Rep analysis to evaluate components, connections, interface types, and mating clearances. It is complemented by code execution checks, geometric validity checks, and VLM-based functional assessment. The authors evaluate 16 models. In Table 2, the best-performing model achieves an execution success rate of 99.06% and a joint success rate of 57.55%. The main contribution is to extend CAD evaluation beyond global shape similarity to explicit component relationships and interface constraints.

Critical strengths

The evaluation targets meaningful engineering constraints, and SIE provides a more diagnostic framework than a global shape score. Component separation, connectivity, interface types, and local clearances are central constraints in multi-component CAD, and a global Chamfer Distance alone cannot adequately characterize them. The paper explicitly represents these aspects through a physical assembly graph and attempts to recover and measure them from STEP geometry (Sections 3.1 and 3.3; Algorithms 1–2). This decomposition can help identify whether a failure arises from missing components, incorrect connections, or incorrect mating dimensions. This is the main technical strength supporting acceptance.

The expert-involved dataset construction and multi-model experiments offer practical value, along with preliminary evidence from human assessment. The authors construct design specifications and CAD references through expert seed modeling, LLM-assisted augmentation, and expert correction, and evaluate 16 general-purpose or code-oriented models (page 5, Section 4.2, and Table 2). Section 4.4 additionally reports two rounds of annotation involving four raters on 20 sampled designs. In Table 3, Gemini 3.1 Pro achieves a design-level Pearson correlation of 0.842. This provides preliminary support for VLM-based functional judgments, although it should not be interpreted as 84.2% accuracy or as direct validation of SIE.

Critical weaknesses

The correctness of the core SIE evaluator lacks independent validation, and the available samples do not sufficiently establish its scope of applicability or the stability of the results. SIE uses programmatic geometric analysis to evaluate component matching, connections, interface types, and clearances. Algorithm 1 uses relative positions, dimensions, and face counts for matching, while Algorithm 2 primarily details rectangular mortise-and-tenon detection (pages 5–6 and 13–14). However, the manuscript does not report matching accuracy, detection errors, or measurement errors against independent expert annotations or manual geometric measurements. The human assessment of 20 sampled designs in Section 4.4 validates the VLM's Functional/Robust judgments and does not substitute for validation of SIE itself. The dataset size and distribution further limit conclusions about applicability: of the 106 instances in Figure 5, 37 are single-component designs, leaving 69 multi-component designs. The connection categories include 49 interlocking, 12 nailing, 6 bonding, and only 2 snap-fit instances (page 7). Although Figure 5 provides some distributional information, it does not sufficiently establish coverage across component counts, interface types, and geometric complexity. The coverage of these categories within the 20 human-assessed designs and the associated statistical uncertainty are also insufficiently characterized. The concern is not that 106 or 20 instances are inherently unacceptable, but that evidence of representativeness and reliability relative to the intended evaluation scope is missing. Consequently, it remains difficult to distinguish model failures from evaluator errors or to determine whether aggregate findings extend to sparsely represented connection types and more complex multi-component designs.

The reported geometric validity results contain inconsistencies with the stated definition and discrepancies between the main text and appendix, limiting verification of the main findings. Section 3.3 explicitly states that Geom. Valid requires successful execution and passing all four geometric checks (page 5, lines 252–256). If the columns are calculated over the same samples with the same denominator, Geom. Valid cannot exceed the pass rate of any necessary condition, including Overlap Free. However, Appendix Table 10 reports Geom. Valid and Overlap Free rates of 20.75% and 19.81% for Qwen2.5-72B, 23.58% and 22.64% for Llama3.1-70B, and 1.89% and 0.94% for Llama3.1-8B, respectively (page 30). In addition, GPT-5.5's Geom. Valid is 66.04% in Table 2 but 68.87% in Table 10; the corresponding values for Gemini 3.1 Pro are 56.60% and 58.49% (pages 7 and 30). These differences cannot be explained solely by rounding to two decimal places. They could result from different runs, denominators, evaluation protocol versions, or incomplete table updates, but the manuscript does not provide enough information to identify the cause. Until these discrepancies are reconciled and instance-level records are provided to reconstruct the aggregate values, questions remain about the verifiability of geometric validity and the joint success results that depend on it. These observations do not, by themselves, establish that the evaluation code is incorrect or that all reported results are invalid.

Questions for authors

The following questions seek evidence supporting the current conclusions. The first two address evaluation validity and coverage, the third addresses numerical consistency, and the fourth clarifies the scope of baseline comparisons.

Agreement between SIE and independent expert assessment: Could experts independently annotate component correspondences, connections, and interface types, and verify clearances through manual geometric measurements, on samples covering different connection categories, correct outputs, and typical failure cases? Please report component-matching accuracy, connection-detection precision and recall, interface-type confusions, and clearance measurement errors. Please also describe the annotation procedure, disagreement resolution, and typical evaluator false positives and false negatives. Clearly distinguish this validation from the existing analysis of VLM functional judgments in Section 4.4. To support verification, please provide the matching weights, thresholds, and decision rules used for each interface category.

Representativeness of the 106 instances and coverage of the 20 human-assessed designs: Beyond the distributions already shown in Figure 5, please characterize the dataset by single-component versus multi-component designs, component count, connection type, and geometric complexity, and explain how these dimensions relate to the intended application scope. How were the 20 designs for human assessment selected, and do they cover the main categories and difficulty levels? Could the authors provide results restricted to multi-component designs and results for the main categories, together with uncertainty estimates, such as appropriate confidence intervals, for key success rates and human-agreement statistics? For categories such as snap-fit, which contains only two instances, please clarify the scope of conclusions supported by the current evidence.

Numerical discrepancies and instance-level verification: How do the authors explain Geom. Valid exceeding Overlap Free in Table 10 and the differences in geometric validity for the same model between Tables 2 and 10? Please specify the sample sets, denominators, evaluation runs, evaluator versions, and final validity rules used for each table. Please provide the instance-level generated code, STEP outputs where generation succeeds, execution status, individual geometric-check flags, and final pass flags underlying the relevant tables, so that readers can investigate the discrepancies and recompute the main results. Status records for failed outputs should also be retained.

Adaptable CAD-specific baselines: Section 4.2 already explains that some prior methods support only single parts or particular command sequences, and that an ArtiCAD implementation was unavailable at the time of evaluation. These limitations should be acknowledged. Given these constraints, are there representative CAD-specific models with available implementations that could reasonably be adapted to the benchmark's input, output, and multi-component requirements? If so, their evaluation would help establish the capability gap between general-purpose or code-oriented LLMs and domain-specific methods. If not, please identify the concrete incompatibilities and required modifications for representative methods, and clarify the scope of the benchmark's current comparative conclusions. This question does not require reimplementing unavailable systems or imposing baselines through unfair task adaptations.

Reviewer_XdAMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

the authors propose MUSE as a benchmark for multi-component Text-to-CAD generation with a focus on component connectivity, interface types, fitting clearances, and assembly structure. The benchmark consists of 106 design instances across manufacturing processes, materials, and connections methods. 16 language models are evaluated for generating executable CAD scripts and producing structurally and geometrically correct assemblies. GPT-6 Astra is the best performing model.

Critical strengths
  1. the combination of B-rep analysis with reference assembly graphs allows specific information about which geometric/structural requirement a generated design fails to satisfy.

  2. the experiments are conducted across 16 language models along with an initial VLM-human agreement study. This provides an experimental scope of the benchmark and insights about failure modes.

Critical weaknesses
  1. the clearance MAE is only over interface-direction pairs for which the predicted and reference measurements are available. This skips undetected or incorrectly localized interfaces. This could make the clearance MAE misleading as models that detect fewer interfaces might achieve low errors.

  2. the definition and computation of joint success is not clear. because there are several constituent measurements that are continuous and i'm not sure how you're converting them to a binary pass/fail decision for the aggregate metric. This would be useful to reliably interpret model rankings.

Questions for authors
  1. have you explored constructing the reference assembly graph directly from the brep geometry instead of an LLM? existing geometry-processing approaches can extract relationships between parts, such as contacts, orientations and clearances. would this make the graph construction process more reproducible?

  2. can you provide measured interfaces per model, tolerance-compliance rates?

  3. which detection processes are used for bonding and snap-fit interfaces?

Reviewer_yFeXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

I first read the paper independently and then discussed its benchmark construction, methodological contribution, and evaluation limitations with ChatGPT. My main concerns were the limited dataset scale and diversity, the reliance on synthetic augmentation, and the insufficient novelty and validation of SIE. I used ChatGPT to organize and polish my review. The final assessment and Clear Reject recommendation reflect my own judgmen

Critical strengths
  1. Relevant evaluation problem. Evaluating assembly connectivity and interface geometry beyond visual similarity is meaningful for multi-component CAD generation.
  2. Executable geometric checks. The use of STEP/B-Rep analysis provides more explicit geometric evidence than purely visual evaluation.
Critical weaknesses
  1. Insufficient benchmark scale and diversity. MUSE contains only 106 designs, including 37 single-body cases. The dataset is dominated by relatively simple furniture and fabrication-oriented objects, with substantial reliance on LLM-generated variants. This is insufficient to support broad claims about multi-component CAD generation.

  2. Limited novelty and insufficient validation of SIE. SIE mainly combines established component matching, graph comparison, and handcrafted interface detection rules. Its generalizability to diverse mechanical interfaces is unclear. More importantly, the paper does not provide direct validation of SIE's detection accuracy against independently annotated interface ground truth or convincing comparisons with alternative evaluation methods. These limitations substantially weaken the paper's methodological contribution.

Questions for authors
  1. Why is a benchmark targeting general multi-component CAD generation limited to 106 designs, including 37 single-body cases?
  2. How well does SIE generalize to mechanical interfaces beyond the predominantly simple connection types represented in the dataset?
  3. Can the authors validate SIE's interface detection and graph recovery accuracy against independent expert annotations and compare it with simpler evaluation baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.