acceptodds
Under review as a conference paper at ICLR 2027

CADBench: A Multimodal Benchmark for AI-Assisted CAD Program Generation

21 views

Abstract

Recovering CAD programs from images or 3D observations is central to AI-assisted design, but progress is difficult to measure because existing evaluations are fragmented across datasets, modalities, and metrics. We introduce CADBench, a unified benchmark for multimodal CAD program generation. CADBench contains 18,000 evaluation samples spanning six benchmark families derived from DeepCAD, Fusion 360, ABC, MCB, and Objaverse; five input modalities including clean meshes, noisy meshes, single-view renders, physically-based renders, and multi-view renders; and metrics for geometric fidelity and code quality, including executability, compactness, and editability. STEP-based families are stratified by B-rep face count and all families are diversity-sampled to support controlled analysis across complexity and object variation. We benchmark eleven CAD-specialized and general-purpose vision-language systems, generating more than 1.4 million CAD programs. CADBench exposes notable variation in model performance across input modalities, geometric complexity, and evaluation criteria. Three recurring failure modes emerge: reconstruction quality degrades with geometric complexity, CAD-specialized models can be brittle under modality shift, and model rankings change across metrics. Together, these results position CADBench as a diagnostic testbed for measuring progress in CAD program generation and multimodal CAD understanding.

Primary area:
datasets and benchmarks
Keywords:
Computer-aided design, editable CAD generation, multimodal benchmarking, geometric reconstruction

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_bc6GOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a benchmark for evaluating CAD program reconstruction from images or 3D observations. The benchmark combines five sources (DeepCAD, Fusion 360, ABC, MCB, and Objaverse) for a total of 18,000 samples. Samples are filtered for diversity within a DINOv3 visual embedding space and stratified by complexity based on B-rep face count. Beyond geometric fidelity, the metrics include code executability, compactness, and editability measures. The authors run a large-scale evaluation of 11 specialized and general-purpose models and report three key findings: reconstruction quality drops with geometric complexity, CAD-specialized models suffer more under modality shift, and models are ranked differently based on different metrics.

Critical strengths

The motivation is well-founded. CAD generation methods have in fact been evaluated on their own datasets and metrics, making cross-paper comparison difficult. The proposed benchmark fills a real gap.

The proposed editability metrics directly reflect real-world CAD practice. For example, Parameterization Ratio encourages parameterized modeling by favoring named variables over hard-coded literals, while DoF Compression penalizes parameter redundancy. This aligns with practitioner expectations in production workflows.

Critical weaknesses

The benchmark's flagship complexity axis is too coarse to support its diagnostic claims. Complexity is tiered by B-rep face count alone, which conflates distinct sources of difficulty (such as topological structure and surface type) that the benchmark never analyzes separately. The benchmark could not pinpoint failure modes: it merely observes performance degradation rather than explaining what causes it.

Questions for authors

Diversity is measured through coverage in a DINOv3 visual embedding space. Is there any evidence that this embedding separates shapes along CAD-relevant axes such as topology or surface type?

DoF Compression rewards programs in which a few root parameters drive many numeric slots. Could a model achieve high compression through semantically meaningless couplings, for example, binding unrelated dimensions to a shared variable? Is there any mechanism to detect such cases?

4 comments

We thank the reviewer for their thoughtful assessment and recognition of the key strengths of our contribution, including the need for a unified CAD benchmark and the practical relevance of our editability metrics. We also appreciate the reviewer’s constructive concerns regarding the benchmark’s complexity measure, failure mode diagnosis, the CAD relevance of DINOv3-based diversity, and the potential for semantically meaningless parameter couplings to inflate DoF Compression. We address each of these points through additional analysis and experiments, which we hope will further strengthen the reviewer’s assessment of our work.

Analyzing complexity beyond face count

Thank you for raising this point. We have added an analysis of how topology and surface characteristics relate to face count and to model performance. We characterize each STEP-derived part along two families of characteristics beyond face count: topology, measured by the first Betti number b1b_1 (the number of independent loops, including through-holes and handles), and surface characteristics, including curved area share, mean absolute curvature times the bounding-box diagonal (at the shape and face levels), feature radii relative to part size, and B-rep statistics such as face types, fillets, and holes.

How do these characteristics relate to performance overall? Pooled over the STEP-derived parts, face count has the strongest Spearman correlation with IoU (ρ=−0.534\rho = -0.534), followed by fillets (−0.43-0.43), b1b_1 (−0.38-0.38), and per-face curvature (−0.33-0.33). These correlations are reported in Appendix C.2.4 of the revised manuscript.

How do they vary across face-count tiers? Across all four STEP-based families, higher tiers exhibit higher b1b_1, more fillets, and finer features (smaller radii relative to part size). Interestingly, they also have a smaller share of curved surface area, as additional faces are predominantly planar and curved features such as fillets and small holes occupy relatively little area. However, the remaining curved faces are more tightly curved, particularly on CAD-Fusion (F) and CAD-All-Ops (A). Each face-count tier therefore contains distinct combinations of geometric characteristics. Per-tier statistics for all families are reported in Appendix C.2.4 (Tables 14 and 15) of the revised manuscript.

Do they affect performance within a tier? Within individual tiers, fillets, b1b_1, and curvature correlate with IoU comparably to face count, whereas holes show no consistent effect. The within-tier correlations are reported in Appendix C.2.4 (Table 16) of the revised manuscript.

Diagnosing the causes of failure

Thank you for raising this point. We have added a failure-mode analysis that attributes losses in geometric fidelity to specific causes, spanning execution failures, structural errors, and dimensional inaccuracies.

Sources of Failure. We decompose the total IoU loss into three categories: (i) execution failures, where the code fails to produce a solid and incurs the full IoU loss; (ii) structural errors, where the code executes but produces incorrect topology (e.g., missing or spurious through-holes, or extra bodies) or proportions (oriented-bounding-box ratios off by at least 0.2); and (iii) dimensional errors, where the topology and overall bounding-box structure are largely correct, but dimensions and finer geometric details differ. Over the STEP-based families (B, F, E, A), these account for 50%, 32%, and 18% of the total IoU loss, respectively. The distribution varies across model families: general-purpose VLMs lose most through execution failures, whereas mesh-input models execute reliably but more frequently produce structural errors.

Execution Failures. Most execution failures arise from factors such as incorrect CadQuery API usage. VLMs often attempt the fillets required by a part, but such code fails to execute more frequently. In contrast, most CAD-specialized models avoid fillets altogether, relying primarily on sketch-and-extrude operations.

Structural Errors. The most common structural errors are missing through-holes, incorrect proportions, and extra bodies. Missing through-holes occur most frequently, while incorrect proportions contribute the largest IoU loss. Mesh-input models more often produce extra bodies, whereas VLMs more often misjudge proportions.

Dimensional Errors. Mesh-input models reproduce proportions accurately (median dimensional error of 0.0–1.2%), whereas VLMs exhibit substantially larger errors (8.5–57.0%). Only 2–8% of their predictions have every dimension within 5%, compared with 68–74% for mesh-input models.

A summary is included in the results section (Section 4), and per-model statistics for all these analyses are reported in Appendix C.3 (Tables 19–21) of the revised manuscript.

Evidence for DINOv3 embeddings separating shapes along CAD-relevant axes

Thank you for raising this point. We have added an analysis in Appendix C.2.6 (Table 18) of the revised manuscript that relates DINOv3 embedding distances to b1b_1, surface type and curvature, feature size, face types, fillets, and holes on all six families. We measure (i) the pairwise correlation between embedding distance and the difference in each characteristic, (ii) neighbor consistency, i.e., the correlation between a part's value and the mean over its 10 nearest DINOv3 neighbors, and (iii) how these differences grow with neighbor rank.

Surface type, curvature, and topology are clearly represented. Pairwise correlations reach ρ=0.45\rho = 0.45 for curved area share; neighbor consistency reaches 0.75–0.93 for curved area share, up to 0.91 for mean absolute curvature, and 0.60–0.87 for b1b_1, and geometric differences grow steadily with embedding distance for every characteristic and family.

Fine details are weaker. Fillets (pairwise ρ\rho = 0.10–0.15) and concave edges are less strongly represented, which is why we use embedding similarity to characterize diversity, and direct geometric metrics to evaluate reconstruction fidelity.

Further details, including per-family values for all 12 characteristics and the distance profiles, are reported in Appendix C.2.6 (Table 18) of the revised manuscript.

Detecting spurious parameter couplings in DoF Compression

We thank the reviewer for this important point. In principle, yes: DoF Compression alone would reward binding unrelated dimensions to one variable. However, our other editability metric, Edit Robustness, provides a natural basis for detecting such cases: meaningless couplings tend to be fragile, since editing the shared variable moves unrelated geometry together and tends to break the part. Building on this, we add an automatic, execution-based decoupling test.

Test. For each variable that earns compression credit, we apply the ±10% Edit Robustness perturbation either to the variable itself, so all of its uses move together (coupled edit), or to only one of its uses (decoupled edit), and check whether the solid keeps its structure. For a genuine design dependency (e.g., a hole sized relative to its plate), the coupled edit keeps the part intact, whereas the decoupled edit tends to break it. The gap G=C−DG = C - D between the coupled (CC) and decoupled (DD) success rates thus measures how much compression credit the geometry confirms. As a baseline, we inject meaningless couplings into the same programs.

Results. Injected couplings are fragile and score G≤0G \le 0 (GG = −0.07 to −0.02), whereas the models' own couplings score clearly above zero for every evaluated VLM (GG = 0.10–0.26), with 95% confidence intervals that exclude both 0 and the injected-coupling baseline. The models' couplings therefore behave unlike meaningless ones, and a considerable share of their compression credit is confirmed by the geometry. These couplings arise from the models' natural coding style. Per-model results, the injected-coupling procedure, and the limitations of the test are reported in Appendix C.6.1 (Table 34) of the revised manuscript.

I thank the authors for their substantial additional analyses, which have significantly strengthened the evaluation protocol. Accordingly, I have increased my overall rating.

We sincerely thank the reviewer for the positive assessment and appreciation of our additional analyses. We are glad that the reviewer was satisfied with our responses.

Reviewer_1aiPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce CadBench, a novel dataset and benchmark for CAD program generation. Compared to earlier datasets, CadBench covers a wider array of tasks, diverse levels of complexity, and includes a suite of evaluation metrics. The authors evaluate a wide range of models on this dataset across multiple input modalities. The benchmark uncovers notable fluctuations in model performance across these modalities and levels of geometric complexity.

Critical strengths

The paper identifies and fills an important gap in CAD evaluation. It is very well written, illustrated, and easy to understand. The model evaluations and ablations on the benchmark uncover interesting model trends and shortcomings, and pave the way for evaluating novel approaches in future work. I believe the paper will be very helpful for the wider CAD generation community.

Critical weaknesses

I am curious why the authors decided not to include deep evaluation metrics, especially since they used DINOv3 embeddings in their filtering pipeline. Wouldn't it make sense to include metrics based on DINOv3 or CLIP/ULIP to strengthen the results? Furthermore, isn't there a bias in other metrics if a model fails to produce valid executable programs? For example, in l.332, IoU is low because outputs rarely compile. Wouldn't it be more robust to resample until a compilable output is retrieved, use that output for evaluation metrics that rely on compiled outputs, and log the retry count as part of the executability metric? This is common in related domains [1,2].

[1]: DeTikZify: Synthesizing Graphics Programs for Scientific Figures and Sketches with TikZ
[2]: TikZero: Zero-Shot Text-Guided Graphics Program Synthesis

Questions for authors

See questions on evaluation metrics in the weakness section.

4 comments

We thank the reviewer for recommending our paper for acceptance and for providing an insightful review that clearly highlights the key strengths of our work. We appreciate the recognition of the importance of the gap our benchmark addresses, the clarity of its presentation, and the insights gained from the model evaluations and ablation studies. We are particularly encouraged that the reviewer sees the benchmark as a valuable resource for the broader CAD generation community and future research. Below, we address the reviewer’s questions and concerns in detail, which we hope will further strengthen the reviewer’s assessment of our work.

Evaluation metrics based on DINOv3, CLIP and ULIP embeddings

Thank you for the suggestion. Our primary motivation for using metrics such as IoU, Chamfer distance (CD) and SIoU was to assess reconstruction accuracy directly. While embedding-based metrics such as DINOv3 capture perceptual similarity and broad shape characteristics, which suits filtering, they do not necessarily provide a reliable measure of geometric fidelity.

For CAD reconstruction, where the objective is to recover the actual geometry rather than merely produce an object that looks similar, these distinctions are important. Two outputs may have similar overall geometric characteristics, while differing substantially in their actual dimensions or geometric features. Nevertheless, we agree that embedding metrics are informative, and we have added them to our evaluation. We now report three embedding-based metrics for every model on all benchmarks (Section 4, Table 3; Appendix C.5.4, Table 31): DINOv3, CLIP and ULIP-2. Each output is aligned to the ground truth and rendered with the same pipeline as our curation renders to obtain the DINOv3 and CLIP based scores, while for ULIP-2, the output and ground truth are sampled as point clouds. The embeddings corresponding to the output and the ground truth are compared by cosine similarity. The table below reports the aggregate embedding similarities alongside IoU under ideal inputs, with failed executions counted as zero.

ModelIoUDINOv3CLIPULIP-2
CADFit0.8550.9110.9720.831
CAD-Recode0.6610.7930.9410.706
CADEvolve0.7080.8340.9500.726
Cadrille0.6720.7940.9420.709
Claude Opus 4.70.4370.8070.9430.693
Gemini 3.1 Pro0.4490.8020.9400.703
GPT-5.40.1740.3190.3680.266
Kimi K2.60.2540.5550.6940.437
Qwen3.5 27B0.0080.0870.1280.051
Qwen3.5 9B0.0000.0000.0000.000
CAD-Coder0.3450.6950.9170.525

All three embedding metrics closely track the overall model ranking (Spearman's ρ\rho = 0.92, 0.95 and 1.00 for DINOv3, CLIP and ULIP-2), identifying the same best- and worst-performing models as IoU. However, this agreement in ranking does not imply that the metrics capture geometric fidelity equally well. The embedding scores are typically more lenient, most of all for the image-input models: Claude Opus 4.7 reaches a DINOv3 similarity of 0.807 at an IoU of only 0.437. Embedding-based metrics thus reward reconstructions that capture the broad appearance or category of a part without accurately recovering its underlying geometry. We add the aggregate embedding scores to the results section of the revised paper (Section 4, Table 3), and the finer-grained scores for each benchmark category in Appendix C.5.4 (Table 31) along with a table that shows median embedding similarity of executed predictions per IoU range in Appendix C.5.4 (Table 30).

Effect of executability failures on geometry metrics and retry-based evaluation

Bias in Metrics. We quantify the effect of evaluating only compilable outputs in Appendix C.7 (Table 35), comparing success-only medians against medians in the situation where zero scores are assigned to failed programs. Success-only evaluation inflates median IoU for every model that fails, with the bias increasing as VSR decreases: by up to 0.075 for CAD-specialized models, 0.120–0.188 for Claude Opus 4.7 and Gemini 3.1 Pro, and 0.316–0.459 for the four lowest-VSR models. Conditioning on success would thus disproportionately reward models for outputs they fail to produce. We therefore include failed executions in geometric fidelity metrics while reporting executability separately through VSR.

We agree that resampling until a compilable output is retrieved is a well-posed protocol as mentioned by the reviewer. We opted for the current protocol of three independent runs for the following reasons.

1. Evaluation objective. CADBench evaluates a model's ability to generate correct CAD programs in one/few attempts, reflecting practical usage and encouraging improvements in generation rather than reliance on repeated sampling. Resampling until success would instead reward models for eventually producing an executable program, obscuring their underlying generation reliability.

2. Evaluation scale and cost. The setting of DeTikZify and TikZero is smaller with evaluation on 1,000 samples across at most two modalities, and resampling is rarely needed there (only 1–2 out of 10 runs for DeTikZify models and 98.5% compilation success for Claude 3 and GPT-4V after the initial Self-Refine Step). CADBench evaluates 18,000 samples (18× more) across five modalities and eleven models, with three independent runs, totaling over 1.4 million generated programs. Moreover, the weakest models require the most retries (e.g., Qwen3.5 9B fails on approximately 80% of samples; Table 10). Resampling until success would therefore substantially increase evaluation time and API costs.

3. Evidence from iterative refinement. We evaluated the closest feasible variant of this protocol in Appendix C.8 with Claude Opus 4.7 in an agentic loop on a balanced 540-sample subset. Refinement raises VSR from 0.839 to 1.000 and IoU from 0.414 to 0.493, at 5.4× the wall-clock cost, but the IoU gain comes almost entirely from repairing programs that failed execution. Improvements in geometric fidelity during iterative refinement were not statistically significant: on samples executable in both settings, all differences in geometric fidelity have 95% confidence intervals that include zero. We also tested with Qwen3.5 9B and found that even with more than ten attempts, improvements in executability diminish, reaching a VSR of only 0.560 after 20 attempts and falling short of eliminating execution failures (Appendix C.8).

I thank the authors for running additional experiments and clarifying the evaluation scheme. I now understand the reasoning of the authors for the chosen evaluation scheme. As such I am keeping my good score and still recommend acceptance.

Thank you for your favorable assessment. We are glad that our response addressed the concerns.

Reviewer_B4poOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces CADBench, a benchmark for reconstructing executable CAD programs from image and mesh observations. It has six source families and provides five input conditions for each object, including noisy meshes and different renderings. The evaluation includes geometric fidelity, executability, compactness, and parameter-level editability measures. Experiments examine how performance changes with geometry, input conditions, and the evaluation metric. Additional studies investigate sampling, training-domain mismatch, and iterative refinement. The main contribution is a common evaluation framework and the diagnostic evidence it provides.

Critical strengths

The paper provides a useful common basis for comparing CAD generators, With a lot of datapoints. The same objects are evaluated under several observation conditions, while the source families cover a wider range of geometry than a single reconstruction dataset. This makes it possible to examine whether a method’s apparent strength depends on familiar inputs or relatively simple parts. The contribution is the coordinated evaluation and the comparisons it enables, rather than the scale of the dataset alone.

The analysis gives meaningful insight into model behavior. The paper goes beyond reporting which system scores highest. It investigates when surface and volume metrics disagree, shows that training-domain alignment can change a model’s preferred input condition, and distinguishes repairing invalid code from improving reconstructed geometry. These results help explain what current systems can do and which apparent improvements should be interpreted cautiously.

Critical weaknesses

The paper does not specify whether Edit Robustness is measured only on initially valid programs or how outcomes from multiple parameter perturbations are combined.

Questions for authors

Please clarify which outputs enter the Edit Robustness calculation, how initially invalid programs are handled, and how outcomes are aggregated across parameters and perturbations.

1 comment

We thank the reviewer for recommending our paper for acceptance and for providing a positive assessment of our method and its contributions. We appreciate the recognition of the benchmark’s value as a large-scale, common basis for comparing CAD generation methods across diverse geometry sources and observation conditions through coordinated evaluation. We are also encouraged that our analyses offer insights beyond aggregate performance, highlighting metric discrepancies and the complementary aspects of code execution and geometric reconstruction quality in characterizing the capabilities and limitations of current CAD generation systems. Below, we clarify the points raised by the reviewer.

Clarification about Edit Robustness

For edit robustness, invalid programs are handled by assigning the edit robustness value to zero. The values in Table 2 (Section 4) of the revised manuscript report this value. However, we also provide in Appendix C.7 an additional experiment in which we only consider the valid programs, and examine the effect this has on edit robustness, with more interpretation of the results (Table 36). We have made the definition clearer in the main paper and in the appendix section for this.

When there are multiple parameter perturbations, for each parameter perturbation we record the score (0 if not successfully executed or if not watertight, 1 if successfully executed and watertight) and aggregate by taking the mean over the different perturbations to obtain the edit robustness.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of our Rebuttal

We thank all reviewers for their thoughtful feedback. Two reviewers recommend acceptance, highlighting the value of CADBench as a large-scale, common basis for comparing CAD generation methods, the importance of the gap it addresses, and the insights provided by its model evaluations and analyses. The remaining reviewer raised concerns about the complexity axis, failure-mode diagnosis, the CAD relevance of the DINOv3-based diversity measure, and the possibility that DoF Compression rewards semantically meaningless couplings. We have sought to address these concerns through clarifications, additional experiments, and analyses. Below, we summarize the key updates.

1. Geometric Fidelity and Evaluation Protocol

  • Embedding-based metrics (Section 4, Table 3; Appendix C.5.4): We now report DINOv3, CLIP, and ULIP-2 similarity scores for every model. These metrics closely track the IoU-based model rankings but are more lenient, rewarding reconstructions that capture a part's broad appearance without accurately recovering its underlying geometry. This further motivates our use of IoU, CD, and SIoU to directly assess geometric reconstruction accuracy.
  • Effect of failed executions (Appendix C.7): Evaluating only compilable outputs inflates the geometric fidelity scores of all models that experience execution failures, with increasingly large effects as executability decreases. We therefore include failed executions in the fidelity metrics while reporting executability separately through VSR.
  • Resampling until success (Appendix C.8): We retain three independent runs because CADBench aims to evaluate whether a model can generate correct CAD programs in one or a few attempts, and resampling at the scale of CADBench would substantially increase evaluation costs. For models such as Qwen3.5 9B, even with more than ten attempts, improvements in executability diminish, reaching a VSR of only 0.560 after 20 attempts and falling short of eliminating execution failures.

2. Editability Metrics

  • Edit Robustness definition (Section 3; Appendices C.6–C.7): Programs that are invalid before perturbation receive a score of zero, and each program's score is computed as the mean across its parameter perturbations. We have made this scoring procedure explicit in the main paper and additionally report a valid-only variant for comparison.
  • Meaningless couplings in DoF Compression (Appendix C.6.1): We introduce an Edit Robustness-based decoupling test, using injected meaningless couplings as a baseline. Injected couplings are fragile under parameter perturbations, whereas the models' own couplings are not, suggesting that the latter are not simply semantically meaningless parameter dependencies.

3. Complexity and Diversity

  • Complexity beyond face count (Appendix C.2.4): We characterize parts using additional topological and surface characteristics, including curvature, feature size, face types, fillets, and holes. Although face count remains the strongest overall correlate of IoU, fillets, topology, and curvature show comparable importance within individual face-count tiers, so difficulty within a tier depends on more than face count.
  • CAD relevance of DINOv3 embeddings (Appendix C.2.6): We find that surface type, curvature, and topology are clearly represented in the embedding space, whereas finer geometric details, such as fillets, are less strongly captured. These findings provide additional evidence supporting the use of DINOv3 embeddings for diversity-based sampling.

4. Failure-Mode Diagnosis (Section 4; Appendix C.3)

  • Sources of failure: We attribute the IoU deficit to three sources: execution failures, structural errors, and dimensional errors.
  • Execution and structural errors: Execution failures arise primarily from issues such as incorrect CadQuery API usage. Missing through-holes as well as extra bodies are among the most common structural errors, while incorrect proportions are the most costly.
  • Dimensional errors: Models operating on mesh inputs reproduce dimensions accurately, whereas VLMs struggle to infer dimensions from images.
  • Systematic failures: Failures concentrate on a shared subset of difficult parts across models. This finding supports reporting performance by benchmark family rather than relying exclusively on aggregate metrics.

Overall, these additions provide a more comprehensive characterization of CADBench's evaluation metrics, the factors influencing model performance, and the limitations of existing CAD generation approaches. We believe the new experiments and analyses address the reviewers' concerns and further strengthen CADBench as a rigorous, large-scale benchmark for evaluating CAD generation across geometric fidelity, editability, and complexity. We thank the reviewers again for their constructive feedback, which has helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.