- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MUSE, a benchmark for CAD generation from structured design specifications, with a focus on assembly structure and interface geometry in multi-component models. The dataset contains 106 design instances constructed through expert seed modeling, LLM-assisted augmentation, and expert review. Each input includes a natural-language design description and a physical assembly graph; the model generates a CadQuery program that exports a STEP file. The core method, Structure and Interface Evaluation (SIE), combines reference-component matching with direct B-Rep analysis to evaluate components, connections, interface types, and mating clearances. It is complemented by code execution checks, geometric validity checks, and VLM-based functional assessment. The authors evaluate 16 models. In Table 2, the best-performing model achieves an execution success rate of 99.06% and a joint success rate of 57.55%. The main contribution is to extend CAD evaluation beyond global shape similarity to explicit component relationships and interface constraints.
Critical strengths
The evaluation targets meaningful engineering constraints, and SIE provides a more diagnostic framework than a global shape score. Component separation, connectivity, interface types, and local clearances are central constraints in multi-component CAD, and a global Chamfer Distance alone cannot adequately characterize them. The paper explicitly represents these aspects through a physical assembly graph and attempts to recover and measure them from STEP geometry (Sections 3.1 and 3.3; Algorithms 1–2). This decomposition can help identify whether a failure arises from missing components, incorrect connections, or incorrect mating dimensions. This is the main technical strength supporting acceptance.
The expert-involved dataset construction and multi-model experiments offer practical value, along with preliminary evidence from human assessment. The authors construct design specifications and CAD references through expert seed modeling, LLM-assisted augmentation, and expert correction, and evaluate 16 general-purpose or code-oriented models (page 5, Section 4.2, and Table 2). Section 4.4 additionally reports two rounds of annotation involving four raters on 20 sampled designs. In Table 3, Gemini 3.1 Pro achieves a design-level Pearson correlation of 0.842. This provides preliminary support for VLM-based functional judgments, although it should not be interpreted as 84.2% accuracy or as direct validation of SIE.
Critical weaknesses
The correctness of the core SIE evaluator lacks independent validation, and the available samples do not sufficiently establish its scope of applicability or the stability of the results. SIE uses programmatic geometric analysis to evaluate component matching, connections, interface types, and clearances. Algorithm 1 uses relative positions, dimensions, and face counts for matching, while Algorithm 2 primarily details rectangular mortise-and-tenon detection (pages 5–6 and 13–14). However, the manuscript does not report matching accuracy, detection errors, or measurement errors against independent expert annotations or manual geometric measurements. The human assessment of 20 sampled designs in Section 4.4 validates the VLM's Functional/Robust judgments and does not substitute for validation of SIE itself. The dataset size and distribution further limit conclusions about applicability: of the 106 instances in Figure 5, 37 are single-component designs, leaving 69 multi-component designs. The connection categories include 49 interlocking, 12 nailing, 6 bonding, and only 2 snap-fit instances (page 7). Although Figure 5 provides some distributional information, it does not sufficiently establish coverage across component counts, interface types, and geometric complexity. The coverage of these categories within the 20 human-assessed designs and the associated statistical uncertainty are also insufficiently characterized. The concern is not that 106 or 20 instances are inherently unacceptable, but that evidence of representativeness and reliability relative to the intended evaluation scope is missing. Consequently, it remains difficult to distinguish model failures from evaluator errors or to determine whether aggregate findings extend to sparsely represented connection types and more complex multi-component designs.
The reported geometric validity results contain inconsistencies with the stated definition and discrepancies between the main text and appendix, limiting verification of the main findings. Section 3.3 explicitly states that Geom. Valid requires successful execution and passing all four geometric checks (page 5, lines 252–256). If the columns are calculated over the same samples with the same denominator, Geom. Valid cannot exceed the pass rate of any necessary condition, including Overlap Free. However, Appendix Table 10 reports Geom. Valid and Overlap Free rates of 20.75% and 19.81% for Qwen2.5-72B, 23.58% and 22.64% for Llama3.1-70B, and 1.89% and 0.94% for Llama3.1-8B, respectively (page 30). In addition, GPT-5.5's Geom. Valid is 66.04% in Table 2 but 68.87% in Table 10; the corresponding values for Gemini 3.1 Pro are 56.60% and 58.49% (pages 7 and 30). These differences cannot be explained solely by rounding to two decimal places. They could result from different runs, denominators, evaluation protocol versions, or incomplete table updates, but the manuscript does not provide enough information to identify the cause. Until these discrepancies are reconciled and instance-level records are provided to reconstruct the aggregate values, questions remain about the verifiability of geometric validity and the joint success results that depend on it. These observations do not, by themselves, establish that the evaluation code is incorrect or that all reported results are invalid.
Questions for authors
The following questions seek evidence supporting the current conclusions. The first two address evaluation validity and coverage, the third addresses numerical consistency, and the fourth clarifies the scope of baseline comparisons.
Agreement between SIE and independent expert assessment: Could experts independently annotate component correspondences, connections, and interface types, and verify clearances through manual geometric measurements, on samples covering different connection categories, correct outputs, and typical failure cases? Please report component-matching accuracy, connection-detection precision and recall, interface-type confusions, and clearance measurement errors. Please also describe the annotation procedure, disagreement resolution, and typical evaluator false positives and false negatives. Clearly distinguish this validation from the existing analysis of VLM functional judgments in Section 4.4. To support verification, please provide the matching weights, thresholds, and decision rules used for each interface category.
Representativeness of the 106 instances and coverage of the 20 human-assessed designs: Beyond the distributions already shown in Figure 5, please characterize the dataset by single-component versus multi-component designs, component count, connection type, and geometric complexity, and explain how these dimensions relate to the intended application scope. How were the 20 designs for human assessment selected, and do they cover the main categories and difficulty levels? Could the authors provide results restricted to multi-component designs and results for the main categories, together with uncertainty estimates, such as appropriate confidence intervals, for key success rates and human-agreement statistics? For categories such as snap-fit, which contains only two instances, please clarify the scope of conclusions supported by the current evidence.
Numerical discrepancies and instance-level verification: How do the authors explain Geom. Valid exceeding Overlap Free in Table 10 and the differences in geometric validity for the same model between Tables 2 and 10? Please specify the sample sets, denominators, evaluation runs, evaluator versions, and final validity rules used for each table. Please provide the instance-level generated code, STEP outputs where generation succeeds, execution status, individual geometric-check flags, and final pass flags underlying the relevant tables, so that readers can investigate the discrepancies and recompute the main results. Status records for failed outputs should also be retained.
Adaptable CAD-specific baselines: Section 4.2 already explains that some prior methods support only single parts or particular command sequences, and that an ArtiCAD implementation was unavailable at the time of evaluation. These limitations should be acknowledged. Given these constraints, are there representative CAD-specific models with available implementations that could reasonably be adapted to the benchmark's input, output, and multi-component requirements? If so, their evaluation would help establish the capability gap between general-purpose or code-oriented LLMs and domain-specific methods. If not, please identify the concrete incompatibilities and required modifications for representative methods, and clarify the scope of the benchmark's current comparative conclusions. This question does not require reimplementing unavailable systems or imposing baselines through unfair task adaptations.