- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
BenchCAD is a synthetic benchmark for programmatic CAD. It contains 17,900 execution-verified CadQuery programs across 106 part families. Experts wrote the family generators. Selected dimensions or relations in 52 families follow industrial standards or an open specification. The data also include 2,400 original paired QA items and 748 editing pairs. The tasks cover generation from rendered views, QA from images or code, and instruction-based code editing. The main contribution is a practical test of CAD understanding beyond overall shape similarity. The benchmark includes more complex modeling operations and tests changes to existing designs. After answerability filtering, the mean code-image QA gap falls from 0.278 to 0.116. The ood-rl model reaches an IoU of 0.757 on seen families and 0.464 on unseen ones. These results show useful research directions. They concern the synthetic setting used here; transfer to real engineering repositories is not established.
Critical strengths
- The benchmark addresses useful problems in CAD. CAD research needs tests of parameter understanding and design modification. BenchCAD brings these into the same setting as code generation. It covers operations such as helical sweeps and twist extrusion, alongside common construction steps. I particularly value the editing task. Changing an existing design is a common need, and whole-shape similarity alone can miss a failed local edit. The metrics still have limits, but the task design gives the community something useful to build on. (Section 3; Tables 17, 21)
- The resource supports controlled experiments. The family generators make it possible to vary parameters and study the effect of training choices. In Table 4, reinforcement learning raises unseen-family IoU from 0.397 to 0.464. The remaining gap to 0.757 on seen families is worth studying. I also appreciate the QA answerability audit. The authors checked an important confound and narrowed their interpretation. These are useful contributions within a controlled synthetic benchmark. (Table 4; Appendices E, F, L)
Critical weaknesses
- The data distribution needs a fuller account. Figure 1 gives the family taxonomy. Appendix E describes approximate tier balance within subfamilies. Appendix L gives original QA type proportions. Table 17 lists counts for all five edit types. These are useful summaries. They do not fully describe the samples behind each result. I did not find counts by family/subfamily and tier for each split in one place. The sampled parameters and construction variants also need a summary. The 46 distinct APIs show breadth, but not frequency or combinations. Tables 12 and 14 give some evaluation support counts. Table 12 has support of only 2 for twistExtrude and 4 for sweep+helix. These refer to that evaluation, not the full dataset. Rare-operation results need care. The filtered QA size and family/type mix are not reported. The original 2,400-item count cannot describe this subset. Editing also needs a family-by-type breakdown. I cannot yet tell how much overall scores reflect common cases. Counts and a comparison of sample-averaged and family-averaged results would help. Existing metadata and predictions should support most of this. (Appendices E, F, L, P; Tables 12, 14, 17) This does not prove poor coverage. It limits the evidence for it. Standard references constrain selected dimensions or relations. They do not establish a representative industrial distribution. The family holdout also stays within the same generation process. (Appendices A, D)
- The reported scores need to be easier to verify. For Gemini 3.1 Pro thinking in Table 11, 0.60 x IoU + 0.20 x ess gives 0.35376. This exceeds the reported total of 0.3457. The values would conflict if they use the same samples and averaging, with nonnegative remaining terms. An execution-rate factor or conditional averages might explain it. A worked calculation would resolve this. (Section 4.3; Table 11) Appendix E flags 351 instances with declared holes lacking matching geometric evidence. The authors corrected the generators. Please clarify which reported evaluations use affected instances and quantify the effect on operation scores. This audit alone does not invalidate the existing geometric comparisons. (Appendix E)
Questions for authors
Q1 What distributions do the release and evaluation splits actually have? Please give counts by family/subfamily and tier, operation support, and a compact summary of sampled parameters. Identify the ten held-out families and their evaluation counts. Please explain how samples are weighted in the overall results. Would averaging equally over families change the main comparisons? Q2 What do the QA and editing subsets cover? Please give the retained QA count and its family/type mix, alongside the original bank. How were expert disagreements resolved? For editing, Table 17 already gives T1-T5 totals. A family-by-type breakdown would show whether the harder edits cover many families or only a few. Q3 Could you show the full score calculation for one row of Table 11? Please use Gemini 3.1 Pro thinking. Explain execution-rate weighting, conditional averages, and CDscore/HDscore normalization. If correction is needed, does it change the main comparisons? Q4 How much do the hole-audit corrections affect operation scores? How many affected instances are in each evaluation? Were labels corrected? If possible, estimate score changes from label correction or exclusion using stored predictions.
3 comments
Response to Author Rebuttal
Thank you for the response.
The added statistics address most of my concerns about the data distribution. The conditional IoU explanation also clarifies why my original calculation did not match. Please separate the 1.25 reweighting adjustment from the essential-operation contribution. Also, if the QA update only filters questions and fixes the scorer, why can the stored code-side predictions not be rescored?
My recommendation remains weak acceptance. I see value in this benchmark for CAD research.
Thank you for the follow-up and the positive assessment.
1.25 reweighting vs. essential-operation term. The 0.111 in the last row of our table splits into two parts:
- Essential-operation term: 0.098. 98 of the 156 parts in families with a specification pass (0.20 × 98 / 200).
- 1.25 reweighting on the 44 parts of the 23 families without a specification: 0.013.
With the four geometry terms (0.235), this gives the total of 0.3457.
Stored predictions. They can be rescored, and we did this as a check. On the retained questions they cover, the stored predictions of all ten models give a gap of 9.9 points, against 10.0 from the new requests in Table 2. Table 2 uses new requests because, for several models, the stored predictions cover fewer than 100 of the 200 parts. After filtering, the remaining sample is small and the scores fluctuate, so we asked the models again on the full set. The four discontinued models could not be asked again, so their code cells are empty; their stored code predictions score .793 (GPT-5.3), .792 (GPT-5.3 thinking), .651 (Moonshot 8k) and .660 (Moonshot 128k). Since both computations give the same gap, we keep Table 2 as it is.
Thank you for the detailed review. The numbers you asked for are in a new Appendix V (Tables 22–27). We answer your questions in order.
Q1 (W1). Data distribution and the sample behind each result
Release. Families have 86–251 parts each (median 171), and the tiers hold 6,086 / 6,078 / 5,736 parts (easy / medium / hard). Parameters are sampled from a typed schema per family, with wider ranges in harder tiers and standard tables where they apply (Appendix E). Table 26 lists per-family counts by tier for every split.
Operations (Table 24). For example, chamfer appears in 28.0% of parts, revolve in 15.8%, loft in 7.7%, helical sweep in 2.3% and twistExtrude in 0.9%. There are 361 distinct operation combinations, and the 15 most common cover 21.6% of parts. Rare operations do need care: shell and twistExtrude come from one or two families (Appendix V.3, Table 12 caption).
Samples (Table 23). The frontier rows of Table 11 and Tables 13–14 use a 200-part evaluation set covering all 106 families. It uses different seeds and does not overlap the release (§3.1). The image-answerable columns of Table 2 use all 738 retained questions (198 parts). Table 15 uses the evaluation set plus two disjoint 200-part samples of the release. Table 3 and Figures 4, 9, 10 use the 748 BenchCAD-Edit pairs (Figure 11 a 100-pair subset). Overall scores are sample averages. The evaluation set has 62 / 70 / 68 parts per tier and 87 of the 106 families contribute exactly two parts, so sample and family averages are close.
Held-out families. Table 25 lists the ten families, with 1,678 parts in the release and 18 in the evaluation set.
Q2. QA and editing subsets
For this response we went through the QA subset again and removed questions that depend on how a part was modelled rather than on its shape (base plane, extrusion axis, whether a given operation was used). It keeps 738 of the 2,400 questions, on 198 parts and all 106 families (2–16 per family): 272 yes/no questions on shape and features, 372 family-specific parameter questions and 94 relative comparisons. By type, that is all 320 ratio questions, 418 of the 1,742 integer questions and none of the 338 absolute-dimension questions. GPT-5.3 (both variants) and the two Moonshot models have been discontinued, so Table 2 only has their image scores. For the other six models the gap is 10.0 points, against 28.2 on the original bank. During the revision we also fixed a sign bug in the QA scorer (answers involving negative values were scored zero even when correct), which makes the scores more accurate, and we rescored all QA results (Tables 2, 9, 10). When experts disagreed on answerability, two of them discussed it and a third joined if they could not agree.
For editing, every family has 5–12 pairs. T1–T5 cover 86 / 72 / 94 / 94 / 106 families, and 40 families have all five types (Table 26). We agree on representativeness and state this limitation in Appendix A.
Q3. Worked calculation for Table 11 (Gemini 3.1 Pro thinking)
The mismatch comes from two things the caption did not say. The IoU column averages executed samples only (163 of 200), and the total uses a pass/fail indicator p instead of the ess_score column. The total is the mean over all 200 samples of s = 0.60 IoU + 0.20 p + 0.10 F1 + 0.05 CD_s + 0.05 HD_s, where p = 1 if the program passes its family's essential-operation check, CD_s = clip((0.2 − CD)/0.199, 0, 1) and HD_s = clip((0.5 − HD)/0.45, 0, 1) on unit-normalized solids. A program that fails to execute gets zero for IoU, CD_s and HD_s but keeps F1 and p, which come from the code. For the 23 families without an essential-operation specification, p is dropped and the other weights are scaled by 1.25.
The first four terms add up to 0.235, and the essential-operation term gives the remaining 0.111. It is below 0.20 × 0.705 because p is pass/fail, while the ess_score column is a recall. The Table 11 caption gives these definitions.
Q4. Hole audit
None of the 351 flagged parts is in the evaluation set or the QA bank, because those use different seeds. The same audit on the 200 evaluation parts flags five, and we kept their labels. The essential-operation score never reads the reference program, so it does not change. Feature-F1 changes by at most 0.025 per model, the Table 11 total by at most 0.003, and hole recall by at most 3.8 points. BenchCAD-Edit was generated separately: 6 of its 303 source programs overlap the release or the evaluation set and none of them is flagged, and the other 297 were not covered by the audit (Appendix E).