- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces a benchmark for evaluating CAD program reconstruction from images or 3D observations. The benchmark combines five sources (DeepCAD, Fusion 360, ABC, MCB, and Objaverse) for a total of 18,000 samples. Samples are filtered for diversity within a DINOv3 visual embedding space and stratified by complexity based on B-rep face count. Beyond geometric fidelity, the metrics include code executability, compactness, and editability measures. The authors run a large-scale evaluation of 11 specialized and general-purpose models and report three key findings: reconstruction quality drops with geometric complexity, CAD-specialized models suffer more under modality shift, and models are ranked differently based on different metrics.
Critical strengths
The motivation is well-founded. CAD generation methods have in fact been evaluated on their own datasets and metrics, making cross-paper comparison difficult. The proposed benchmark fills a real gap.
The proposed editability metrics directly reflect real-world CAD practice. For example, Parameterization Ratio encourages parameterized modeling by favoring named variables over hard-coded literals, while DoF Compression penalizes parameter redundancy. This aligns with practitioner expectations in production workflows.
Critical weaknesses
The benchmark's flagship complexity axis is too coarse to support its diagnostic claims. Complexity is tiered by B-rep face count alone, which conflates distinct sources of difficulty (such as topological structure and surface type) that the benchmark never analyzes separately. The benchmark could not pinpoint failure modes: it merely observes performance degradation rather than explaining what causes it.
Questions for authors
Diversity is measured through coverage in a DINOv3 visual embedding space. Is there any evidence that this embedding separates shapes along CAD-relevant axes such as topology or surface type?
DoF Compression rewards programs in which a few root parameters drive many numeric slots. Could a model achieve high compression through semantically meaningless couplings, for example, binding unrelated dimensions to a shared variable? Is there any mechanism to detect such cases?
4 comments
Evidence for DINOv3 embeddings separating shapes along CAD-relevant axes
Thank you for raising this point. We have added an analysis in Appendix C.2.6 (Table 18) of the revised manuscript that relates DINOv3 embedding distances to , surface type and curvature, feature size, face types, fillets, and holes on all six families. We measure (i) the pairwise correlation between embedding distance and the difference in each characteristic, (ii) neighbor consistency, i.e., the correlation between a part's value and the mean over its 10 nearest DINOv3 neighbors, and (iii) how these differences grow with neighbor rank.
Surface type, curvature, and topology are clearly represented. Pairwise correlations reach for curved area share; neighbor consistency reaches 0.75–0.93 for curved area share, up to 0.91 for mean absolute curvature, and 0.60–0.87 for , and geometric differences grow steadily with embedding distance for every characteristic and family.
Fine details are weaker. Fillets (pairwise = 0.10–0.15) and concave edges are less strongly represented, which is why we use embedding similarity to characterize diversity, and direct geometric metrics to evaluate reconstruction fidelity.
Further details, including per-family values for all 12 characteristics and the distance profiles, are reported in Appendix C.2.6 (Table 18) of the revised manuscript.
Detecting spurious parameter couplings in DoF Compression
We thank the reviewer for this important point. In principle, yes: DoF Compression alone would reward binding unrelated dimensions to one variable. However, our other editability metric, Edit Robustness, provides a natural basis for detecting such cases: meaningless couplings tend to be fragile, since editing the shared variable moves unrelated geometry together and tends to break the part. Building on this, we add an automatic, execution-based decoupling test.
Test. For each variable that earns compression credit, we apply the ±10% Edit Robustness perturbation either to the variable itself, so all of its uses move together (coupled edit), or to only one of its uses (decoupled edit), and check whether the solid keeps its structure. For a genuine design dependency (e.g., a hole sized relative to its plate), the coupled edit keeps the part intact, whereas the decoupled edit tends to break it. The gap between the coupled () and decoupled () success rates thus measures how much compression credit the geometry confirms. As a baseline, we inject meaningless couplings into the same programs.
Results. Injected couplings are fragile and score ( = −0.07 to −0.02), whereas the models' own couplings score clearly above zero for every evaluated VLM ( = 0.10–0.26), with 95% confidence intervals that exclude both 0 and the injected-coupling baseline. The models' couplings therefore behave unlike meaningless ones, and a considerable share of their compression credit is confirmed by the geometry. These couplings arise from the models' natural coding style. Per-model results, the injected-coupling procedure, and the limitations of the test are reported in Appendix C.6.1 (Table 34) of the revised manuscript.
I thank the authors for their substantial additional analyses, which have significantly strengthened the evaluation protocol. Accordingly, I have increased my overall rating.
We sincerely thank the reviewer for the positive assessment and appreciation of our additional analyses. We are glad that the reviewer was satisfied with our responses.
We thank the reviewer for their thoughtful assessment and recognition of the key strengths of our contribution, including the need for a unified CAD benchmark and the practical relevance of our editability metrics. We also appreciate the reviewer’s constructive concerns regarding the benchmark’s complexity measure, failure mode diagnosis, the CAD relevance of DINOv3-based diversity, and the potential for semantically meaningless parameter couplings to inflate DoF Compression. We address each of these points through additional analysis and experiments, which we hope will further strengthen the reviewer’s assessment of our work.
Analyzing complexity beyond face count
Thank you for raising this point. We have added an analysis of how topology and surface characteristics relate to face count and to model performance. We characterize each STEP-derived part along two families of characteristics beyond face count: topology, measured by the first Betti number b1 (the number of independent loops, including through-holes and handles), and surface characteristics, including curved area share, mean absolute curvature times the bounding-box diagonal (at the shape and face levels), feature radii relative to part size, and B-rep statistics such as face types, fillets, and holes.
How do these characteristics relate to performance overall? Pooled over the STEP-derived parts, face count has the strongest Spearman correlation with IoU (ρ=−0.534), followed by fillets (−0.43), b1 (−0.38), and per-face curvature (−0.33). These correlations are reported in Appendix C.2.4 of the revised manuscript.
How do they vary across face-count tiers? Across all four STEP-based families, higher tiers exhibit higher b1, more fillets, and finer features (smaller radii relative to part size). Interestingly, they also have a smaller share of curved surface area, as additional faces are predominantly planar and curved features such as fillets and small holes occupy relatively little area. However, the remaining curved faces are more tightly curved, particularly on CAD-Fusion (F) and CAD-All-Ops (A). Each face-count tier therefore contains distinct combinations of geometric characteristics. Per-tier statistics for all families are reported in Appendix C.2.4 (Tables 14 and 15) of the revised manuscript.
Do they affect performance within a tier? Within individual tiers, fillets, b1, and curvature correlate with IoU comparably to face count, whereas holes show no consistent effect. The within-tier correlations are reported in Appendix C.2.4 (Table 16) of the revised manuscript.
Diagnosing the causes of failure
Thank you for raising this point. We have added a failure-mode analysis that attributes losses in geometric fidelity to specific causes, spanning execution failures, structural errors, and dimensional inaccuracies.
Sources of Failure. We decompose the total IoU loss into three categories: (i) execution failures, where the code fails to produce a solid and incurs the full IoU loss; (ii) structural errors, where the code executes but produces incorrect topology (e.g., missing or spurious through-holes, or extra bodies) or proportions (oriented-bounding-box ratios off by at least 0.2); and (iii) dimensional errors, where the topology and overall bounding-box structure are largely correct, but dimensions and finer geometric details differ. Over the STEP-based families (B, F, E, A), these account for 50%, 32%, and 18% of the total IoU loss, respectively. The distribution varies across model families: general-purpose VLMs lose most through execution failures, whereas mesh-input models execute reliably but more frequently produce structural errors.
Execution Failures. Most execution failures arise from factors such as incorrect CadQuery API usage. VLMs often attempt the fillets required by a part, but such code fails to execute more frequently. In contrast, most CAD-specialized models avoid fillets altogether, relying primarily on sketch-and-extrude operations.
Structural Errors. The most common structural errors are missing through-holes, incorrect proportions, and extra bodies. Missing through-holes occur most frequently, while incorrect proportions contribute the largest IoU loss. Mesh-input models more often produce extra bodies, whereas VLMs more often misjudge proportions.
Dimensional Errors. Mesh-input models reproduce proportions accurately (median dimensional error of 0.0–1.2%), whereas VLMs exhibit substantially larger errors (8.5–57.0%). Only 2–8% of their predictions have every dimension within 5%, compared with 68–74% for mesh-input models.
A summary is included in the results section (Section 4), and per-model statistics for all these analyses are reported in Appendix C.3 (Tables 19–21) of the revised manuscript.