acceptodds
Under review as a conference paper at ICLR 2027

BenchCAD: A Comprehensive, Standard-Anchored Benchmark for Programmatic CAD

15 views

Abstract

Industrial Computer-Aided Design (CAD) code generation requires models to produce executable parametric programs from visual or textual inputs. Beyond recognizing the outer shape of a part, this task involves understanding its 3D structure, inferring engineering parameters, and choosing CAD operations that reflect how the part would be designed and manufactured. Despite the promise of Multimodal large language models (MLLMs) for this task, they are rarely evaluated on whether these capabilities jointly hold in realistic industrial CAD settings. We present BenchCAD, a synthetic, standard-anchored benchmark for industrial CAD reasoning. BenchCAD contains 17,900 execution-verified CadQuery programs across 106 industrial part families, including bevel gears, compression springs, twist drills, and other reusable engineering designs. It evaluates models through visual question answering, code question answering, image-to-code generation, and instruction-guided code editing, enabling fine-grained analysis across perception, parametric abstraction, and executable program synthesis. Across 10+ frontier models, BenchCAD shows that current systems often recover coarse outer geometry but fail to produce faithful parametric CAD programs. Common failures include missing fine 3D structure, misinterpreting industrial design parameters, and replacing essential operations such as sweeps, lofts, and twist-extrudes with simpler sketch-and-extrude patterns. Fine-tuning and reinforcement learning improve in-distribution performance, but generalization to unseen part families remains limited. An answerability audit narrows the mean code–image QA gap from 27.8 to 11.6 percentage points, showing that the original comparison partly reflected questions that rendered views cannot answer. BenchCAD supports diagnosis of programmatic CAD capabilities within this synthetic setting. BenchCAD has also been used in publicly reported model evaluations by OpenAI and Anthropic. Released under CC-BY-4.0.

Primary area:
datasets and benchmarks
Keywords:
CAD, VLLM, 3d reconstruction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BcVWOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

BenchCAD is a synthetic benchmark for programmatic CAD. It contains 17,900 execution-verified CadQuery programs across 106 part families. Experts wrote the family generators. Selected dimensions or relations in 52 families follow industrial standards or an open specification. The data also include 2,400 original paired QA items and 748 editing pairs. The tasks cover generation from rendered views, QA from images or code, and instruction-based code editing. The main contribution is a practical test of CAD understanding beyond overall shape similarity. The benchmark includes more complex modeling operations and tests changes to existing designs. After answerability filtering, the mean code-image QA gap falls from 0.278 to 0.116. The ood-rl model reaches an IoU of 0.757 on seen families and 0.464 on unseen ones. These results show useful research directions. They concern the synthetic setting used here; transfer to real engineering repositories is not established.

Critical strengths
  1. The benchmark addresses useful problems in CAD. CAD research needs tests of parameter understanding and design modification. BenchCAD brings these into the same setting as code generation. It covers operations such as helical sweeps and twist extrusion, alongside common construction steps. I particularly value the editing task. Changing an existing design is a common need, and whole-shape similarity alone can miss a failed local edit. The metrics still have limits, but the task design gives the community something useful to build on. (Section 3; Tables 17, 21)
  2. The resource supports controlled experiments. The family generators make it possible to vary parameters and study the effect of training choices. In Table 4, reinforcement learning raises unseen-family IoU from 0.397 to 0.464. The remaining gap to 0.757 on seen families is worth studying. I also appreciate the QA answerability audit. The authors checked an important confound and narrowed their interpretation. These are useful contributions within a controlled synthetic benchmark. (Table 4; Appendices E, F, L)
Critical weaknesses
  1. The data distribution needs a fuller account. Figure 1 gives the family taxonomy. Appendix E describes approximate tier balance within subfamilies. Appendix L gives original QA type proportions. Table 17 lists counts for all five edit types. These are useful summaries. They do not fully describe the samples behind each result. I did not find counts by family/subfamily and tier for each split in one place. The sampled parameters and construction variants also need a summary. The 46 distinct APIs show breadth, but not frequency or combinations. Tables 12 and 14 give some evaluation support counts. Table 12 has support of only 2 for twistExtrude and 4 for sweep+helix. These refer to that evaluation, not the full dataset. Rare-operation results need care. The filtered QA size and family/type mix are not reported. The original 2,400-item count cannot describe this subset. Editing also needs a family-by-type breakdown. I cannot yet tell how much overall scores reflect common cases. Counts and a comparison of sample-averaged and family-averaged results would help. Existing metadata and predictions should support most of this. (Appendices E, F, L, P; Tables 12, 14, 17) This does not prove poor coverage. It limits the evidence for it. Standard references constrain selected dimensions or relations. They do not establish a representative industrial distribution. The family holdout also stays within the same generation process. (Appendices A, D)
  2. The reported scores need to be easier to verify. For Gemini 3.1 Pro thinking in Table 11, 0.60 x IoU + 0.20 x ess gives 0.35376. This exceeds the reported total of 0.3457. The values would conflict if they use the same samples and averaging, with nonnegative remaining terms. An execution-rate factor or conditional averages might explain it. A worked calculation would resolve this. (Section 4.3; Table 11) Appendix E flags 351 instances with declared holes lacking matching geometric evidence. The authors corrected the generators. Please clarify which reported evaluations use affected instances and quantify the effect on operation scores. This audit alone does not invalidate the existing geometric comparisons. (Appendix E)
Questions for authors

Q1 What distributions do the release and evaluation splits actually have? Please give counts by family/subfamily and tier, operation support, and a compact summary of sampled parameters. Identify the ten held-out families and their evaluation counts. Please explain how samples are weighted in the overall results. Would averaging equally over families change the main comparisons? Q2 What do the QA and editing subsets cover? Please give the retained QA count and its family/type mix, alongside the original bank. How were expert disagreements resolved? For editing, Table 17 already gives T1-T5 totals. A family-by-type breakdown would show whether the harder edits cover many families or only a few. Q3 Could you show the full score calculation for one row of Table 11? Please use Gemini 3.1 Pro thinking. Explain execution-rate weighting, conditional averages, and CDscore/HDscore normalization. If correction is needed, does it change the main comparisons? Q4 How much do the hole-audit corrections affect operation scores? How many affected instances are in each evaluation? Were labels corrected? If possible, estimate score changes from label correction or exclusion using stored predictions.

3 comments

Thank you for the detailed review. The numbers you asked for are in a new Appendix V (Tables 22–27). We answer your questions in order.

Q1 (W1). Data distribution and the sample behind each result

Release. Families have 86–251 parts each (median 171), and the tiers hold 6,086 / 6,078 / 5,736 parts (easy / medium / hard). Parameters are sampled from a typed schema per family, with wider ranges in harder tiers and standard tables where they apply (Appendix E). Table 26 lists per-family counts by tier for every split.

Operations (Table 24). For example, chamfer appears in 28.0% of parts, revolve in 15.8%, loft in 7.7%, helical sweep in 2.3% and twistExtrude in 0.9%. There are 361 distinct operation combinations, and the 15 most common cover 21.6% of parts. Rare operations do need care: shell and twistExtrude come from one or two families (Appendix V.3, Table 12 caption).

Samples (Table 23). The frontier rows of Table 11 and Tables 13–14 use a 200-part evaluation set covering all 106 families. It uses different seeds and does not overlap the release (§3.1). The image-answerable columns of Table 2 use all 738 retained questions (198 parts). Table 15 uses the evaluation set plus two disjoint 200-part samples of the release. Table 3 and Figures 4, 9, 10 use the 748 BenchCAD-Edit pairs (Figure 11 a 100-pair subset). Overall scores are sample averages. The evaluation set has 62 / 70 / 68 parts per tier and 87 of the 106 families contribute exactly two parts, so sample and family averages are close.

Held-out families. Table 25 lists the ten families, with 1,678 parts in the release and 18 in the evaluation set.

Q2. QA and editing subsets

For this response we went through the QA subset again and removed questions that depend on how a part was modelled rather than on its shape (base plane, extrusion axis, whether a given operation was used). It keeps 738 of the 2,400 questions, on 198 parts and all 106 families (2–16 per family): 272 yes/no questions on shape and features, 372 family-specific parameter questions and 94 relative comparisons. By type, that is all 320 ratio questions, 418 of the 1,742 integer questions and none of the 338 absolute-dimension questions. GPT-5.3 (both variants) and the two Moonshot models have been discontinued, so Table 2 only has their image scores. For the other six models the gap is 10.0 points, against 28.2 on the original bank. During the revision we also fixed a sign bug in the QA scorer (answers involving negative values were scored zero even when correct), which makes the scores more accurate, and we rescored all QA results (Tables 2, 9, 10). When experts disagreed on answerability, two of them discussed it and a third joined if they could not agree.

For editing, every family has 5–12 pairs. T1–T5 cover 86 / 72 / 94 / 94 / 106 families, and 40 families have all five types (Table 26). We agree on representativeness and state this limitation in Appendix A.

Q3. Worked calculation for Table 11 (Gemini 3.1 Pro thinking)

The mismatch comes from two things the caption did not say. The IoU column averages executed samples only (163 of 200), and the total uses a pass/fail indicator p instead of the ess_score column. The total is the mean over all 200 samples of s = 0.60 IoU + 0.20 p + 0.10 F1 + 0.05 CD_s + 0.05 HD_s, where p = 1 if the program passes its family's essential-operation check, CD_s = clip((0.2 − CD)/0.199, 0, 1) and HD_s = clip((0.5 − HD)/0.45, 0, 1) on unit-normalized solids. A program that fails to execute gets zero for IoU, CD_s and HD_s but keeps F1 and p, which come from the code. For the 23 families without an essential-operation specification, p is dropped and the other weights are scaled by 1.25.

Term (mean over 200 samples)ValueWeightContribution
IoU (0.3546 on executed × 0.815)0.2890.600.173
Feature-F10.0670.100.007
CD_s (0.787 × 0.815)0.6410.050.032
HD_s (0.560 × 0.815)0.4560.050.023
Essential-operation term (p, incl. 1.25 rescaling)0.200.111
Total0.3457

The first four terms add up to 0.235, and the essential-operation term gives the remaining 0.111. It is below 0.20 × 0.705 because p is pass/fail, while the ess_score column is a recall. The Table 11 caption gives these definitions.

Q4. Hole audit

None of the 351 flagged parts is in the evaluation set or the QA bank, because those use different seeds. The same audit on the 200 evaluation parts flags five, and we kept their labels. The essential-operation score never reads the reference program, so it does not change. Feature-F1 changes by at most 0.025 per model, the Table 11 total by at most 0.003, and hole recall by at most 3.8 points. BenchCAD-Edit was generated separately: 6 of its 303 source programs overlap the release or the evaluation set and none of them is flagged, and the other 297 were not covered by the audit (Appendix E).

Reviewer_BcVWOpenReview ↗

Response to Author Rebuttal

Thank you for the response.

The added statistics address most of my concerns about the data distribution. The conditional IoU explanation also clarifies why my original calculation did not match. Please separate the 1.25 reweighting adjustment from the essential-operation contribution. Also, if the QA update only filters questions and fixes the scorer, why can the stored code-side predictions not be rescored?

My recommendation remains weak acceptance. I see value in this benchmark for CAD research.

Thank you for the follow-up and the positive assessment.

1.25 reweighting vs. essential-operation term. The 0.111 in the last row of our table splits into two parts:

  • Essential-operation term: 0.098. 98 of the 156 parts in families with a specification pass (0.20 × 98 / 200).
  • 1.25 reweighting on the 44 parts of the 23 families without a specification: 0.013.

With the four geometry terms (0.235), this gives the total of 0.3457.

Stored predictions. They can be rescored, and we did this as a check. On the retained questions they cover, the stored predictions of all ten models give a gap of 9.9 points, against 10.0 from the new requests in Table 2. Table 2 uses new requests because, for several models, the stored predictions cover fewer than 100 of the 200 parts. After filtering, the remaining sample is small and the scores fluctuate, so we asked the models again on the full set. The four discontinued models could not be asked again, so their code cells are empty; their stored code predictions score .793 (GPT-5.3), .792 (GPT-5.3 thinking), .651 (Moonshot 8k) and .660 (Moonshot 128k). Since both computations give the same gap, we keep Table 2 as it is.

Reviewer_tENROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BenchCAD, a benchmark for evaluating models’ understanding, generation, and editing of parametric CAD. Its central premise is that CAD evaluation should capture operation selection and parametric relationships beyond geometric similarity. The benchmark combines industrially grounded CAD models with generation, question-answering, and editing tasks. Experiments evaluate existing models and investigate whether training on BenchCAD improves CAD capabilities and generalization.

Critical strengths
  1. The benchmark combines CAD generation, paired image/code QA, and instruction-guided editing, allowing complementary aspects of CAD capabilities to be examined.

  2. The family-held-out evaluation and QA answerability audit provide useful checks on generalization and the interpretation of modality comparisons.

Critical weaknesses
  1. Insufficient evidence for the reasoning claims in the primary QA evaluation.

The primary filtered QA subset is insufficiently characterized: the paper does not report its size, the number or proportion of retained questions requiring engineering-standard relations or parameter dependencies, the constraints they cover, or separate performance on these questions. Although Tables 9 and 10 provide capability-level scores, they concern the original, unfiltered question bank. It therefore remains unclear to what extent the aggregate scores in Table 2 support the paper’s claims about industrial parametric reasoning.

  1. The manuscript’s writing quality and internal consistency raise my concerns:

(a) Potentially hallucinated reference. I can not find the reference “Khaled Alrashedy et al. CAD-CodeVerify: Self-verification of vision-language models on CAD code generation. ICLR, 2025.” on Google Scholar and ICLR 2025 paper list.

(b) Table 2 formatting. The caption is placed beside the table, which appears inconsistent with the presentation in the official ICLR 2027 template.

(c) Inconsistent aggregation in Table 12. The table lists 19 operations but reports “Macro recall (16 ops),” without specifying the included operations or any exclusion or merging rules. The reported averages cannot be reproduced from all listed operations, and the OOD-SFT and IID-SFT averages cannot be obtained from any 16-operation subset. The aggregation rule and numerical consistency require clarification.

(d) Missing supplementary evidence. Line 269 points to Appendix Q for a scale-invariance ablation. Table 21 promises saturation curves and complete per-corpus operation lists in Appendix R.

(e) Inconsistent words. For example, the o3 model is referred to as “o3” in Table 2, “OpenAI o3” in Table 3, “openai-o3” in Tables 9–10, and “openai/o3” in Table 11.

Questions for authors
  1. What proportion of the retained QA set requires reasoning about standard relations or parameter dependencies, and how do models perform on these questions?

  2. Could you clarify the issues raised in Weakness 2 and indicate whether any stem from a misunderstanding on my part?

4 comments

Thank you for the careful reading in detail. We answer W1 first, then W2 and your later comment on the author lists.

W1 / Q1. What is in the filtered QA subset, and how do models perform on the parametric questions?

The submission did not describe the subset, and it should have. While preparing this response we went through the subset again and tightened it: we removed questions that depend on how a part was modelled rather than on its shape (base plane, extrusion axis, whether an operation such as revolve was used). The subset has 738 of the 2,400 questions, on 198 parts and all 106 families, and both inputs are evaluated on exactly these questions (Appendix L, Table 27).

Question groupQuestionsImageCodeGap
Shape / feature yes-no272 (37%).772.880.108
Family-specific parameters372 (50%).684.776.092
– on standard-anchored families181 (25%).687.779.093
Relative comparisons of feature dimensions94 (13%).688.795.107
All738.717.817.100

To answer Q1: half of the retained questions ask for a family parameter, for example the ball-to-stem radius ratio of a ball knob, the spring index of a coil spring, or the number of teeth of a gear. 181 of them are on standard-anchored families. Models do worse on these questions than on the shape questions with either input (.684 vs .772 from the image), and the standard-anchored ones are not easier (.687).

Three notes on how these numbers were obtained. First, the subset scores come from new requests, not from the predictions stored in the original run. For several models the stored predictions cover fewer than 100 of the 200 parts, so after filtering the sample would be small and the scores would fluctuate. We therefore asked the models again on all parts. Second, GPT-5.3 (both variants) and the two Moonshot models were discontinued by their providers before the new code requests, so Table 2 only has their image scores and the averages are over the other six models. Third, during the revision we found a sign bug in the QA scorer. Answers involving negative values, such as a signed sum of cut depths, were scored zero even when correct. The scorer now compares magnitudes and uses ratio accuracy for every answer (§4.3), and we rescored all QA results (Tables 2, 9, 10). The gap is 10.0 points on the subset and 28.2 on the original bank (11.6 and 27.8 in the submission, over ten models). The conclusion is the same: most of the gap disappears after filtering, and code input still helps.

We also agree with your main point. Table 2 by itself does not support claims about industrial parametric reasoning, and the revised §5.1 says it only measures the effect of information access. The evidence for parametric reasoning comes from Vision2Code and CodeEdit (Tables 13–14, Figure 5D, the T3 and T5 edits).

W2 / Q2. Writing and consistency.

All five points are correct, and none is a misunderstanding on your side. We fixed each of them.

(a) Reference. The paper we meant is Alrashedy et al., "Generating CAD Code with Vision-Language Models for 3D Designs", ICLR 2025 (CADCodeVerify / CADPrompt, https://openreview.net/forum?id=BLWaTeucYX). Our entry had the wrong first name and title. We then checked the authors, title and venue of every paper in the reference list against the publisher or arXiv page.

(b) Table 2. The caption is above the table, as in the template.

(c) Table 12. The macro row was out of date. Over the 19 listed operations it is 19.7% / 67.6% / 88.0%, and over the 11 advanced operations 0.0% / 62.4% / 86.0%. The caption states how it is computed (unweighted mean over the listed operations). The conclusion does not change.

(d) Appendices Q and R. Scale invariance holds by construction, because both solids are unit-normalized before voxelization (Appendix O). The sentence you cite (line 269 of the submission) says this instead of pointing to an ablation. Table 21 no longer refers to saturation curves or per-corpus lists. Appendix R gives the saturation points, and Table 24 gives the per-operation frequencies for BenchCAD.

(e) Model names. Each model has one name everywhere, the same as in the figure legends (e.g. "OpenAI o3").

Two further corrections from the reference check in (a). The author lists of two entries were wrong. They now read:

  • CAD-Recode (ICCV 2025): Danila Rukhovich, Elona Dupont, Dimitrios Mallis, Kseniya Cherenkova, Anis Kacem, Djamila Aouada.
  • cadrille (ICLR 2026): Maksim Kolodiazhnyi, Denis Tarasov, Dmitrii Zhemchuzhnikov, Alexander Nikulin, Ilya Zisman, Anna Vorontsova, Anton Konushin, Vladislav Kurenkov, Danila Rukhovich.
Reviewer_tENROpenReview ↗

Thanks for the response

Thank you for your response. Could you clarify what “Author lists (your later comment)” refers to? I have not made any comments about the author lists.

Apologies for the confusing heading. It refers to the update of the reference you pointed out in W2(a). While re-checking the reference list, we also found and corrected the author lists of two other entries, CAD-Recode and cadrille. We have fixed the heading in our response.

Thanks for the response. However, I maintain my rating (2) because of the following reasons:

  1. The manuscript contained serious errors in its references. Seems like AI hallucination.
  2. The authors reported incorrect numerical values in the experiment results. I consider this a serious error.
  3. The manuscript contained many writing errors, and some error content in the appendix.
  4. The authors’ rebuttal included the puzzling phrase “Author lists (your later comment)”. I never mentioned author lists in my review, and I think this phrase is may also generated by AI.

Although the authors have corrected these issues, the above reasons leave me with serious reservations about the overall quality of the paper. Taken together, the above reasons also raise a concern that substantial portions of both the manuscript and the rebuttal content may have been AI-generated.

Reviewer_LGdEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents BenchCAD, a synthetic, standards-anchored benchmark for industrial CAD reasoning. It contains 17,900 execution-verified CadQuery programs spanning 106 industrial part families. BenchCAD supports fine-grained analysis of perception, parametric abstraction, and executable program synthesis through four tasks: visual question answering, code question answering, image-to-code generation, and instruction-guided code editing. Experiments on more than 10 models show that current models often recover coarse outer geometry but struggle to produce faithful parametric CAD programs.

Critical strengths
  • The paper finds that current evaluations of CAD code generation do not fully assess the quality of generated CAD models and presents a new benchmark with four task families.
  • The evaluation is comprehensive, and the proposed BenchCAD reveals a gap between apparent geometric similarity and true parametric understanding.
Critical weaknesses
  • The design of four task families is not sufficiently motivated. How do the four task families assess models’ understanding of operations, parameters, constraints, and editable structures, respectively? Further justification is needed.
  • For the vision2code task, the paper evaluates models’ code-generation capabilities using four canonical orthographic views. Are these views sufficient?
Questions for authors
  • How do the four task families assess models’ understanding of operations, parameters, constraints, and editable structures, respectively?
  • For the vision2code task, are the four canonical orthographic views sufficient for evaluating vision2code capability?
2 comments

Thank you for the positive feedback and for the two questions. We answer both below.

W1/Q1. How do the four tasks assess operations, parameters, constraints and editable structure?

Each task keeps a different part of the problem fixed, so when a model fails we can tell which aspect it failed on, instead of only seeing one end-to-end score. We added Table 22 (Appendix V) to make this mapping explicit:

AspectVision2CodeImage / Code QACodeEdit
Operationsessential-op recall, per-op recall (Table 14)L2 operation questionsT2 transforms, T4 add/remove features
ParametersIoU, CD, HD, Feature-F1L3 parameter questionsT1 literal edits, T3 derived values
Constraintsstandard tables in 52 families, coupled parameters in the hard tier (Table 13)ratio questionsT3, T5 coordinated edits
Editable structurenot measured (the program is written from scratch)construction history (code input)the task itself: one change to a valid program

Image QA and Code QA ask the same questions with different inputs, so the gap between them shows what is lost when the model only sees the image.

W2/Q2. Are the four views sufficient for Vision2Code?

Yes. With the same four views and a rendering tool, recent models (Claude Opus 5.5 and GPT-6 Astra) reach a voxel IoU of about 0.96 on BenchCAD (Figure 7, Appendix J). So the four views contain enough information to reconstruct the parts.

Thank you for the response. It would be better to put this mapping into the main paper rather than appendix for better understanding. I would like to maintain my positive rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment

We thank all reviewers for their time and helpful feedback. We are glad that the reviewers see BenchCAD as a useful test of CAD understanding beyond shape similarity (LGdE, BcVW), and that they found the editing task (BcVW), the held-out-family evaluation and the answerability audit (tENR, BcVW) valuable.

We have uploaded a revised PDF. All changes are marked in blue. In summary:

  • Data and evaluation details (tENR, BcVW). A new Appendix V (Tables 22–27) reports per-family counts for every split, operation frequencies, the held-out families, the size and composition of the image-answerable QA subset, and which sample each result is computed on. The Table 11 caption gives the full score definition.
  • Presentation fixes (tENR). We corrected the references, the Table 12 macro row, the appendix pointers and the model names.
  • Task motivation (LGdE). Table 22 maps each task to the aspect of CAD understanding it measures.

We respond to each reviewer separately below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.