acceptodds
Under review as a conference paper at ICLR 2027

MCAD-Bench: Benchmarking CAD Editing on Real Supplier Parts with a Controlled Answer Channel

Abstract

Benchmarks that ask a language model to produce CAD almost always require the answer as a program in a geometry-kernel API and then score the solid that program builds. We show that this choice is part of the measurement, not an implementation detail. CAD systems share features but not syntax, so no canonical API settles the matter, and a representation that moves the result should be controlled rather than ranked. MCAD-Bench is built from 144 real supplier products stored as STEP, with three task families (reconstruction, instruction-guided editing by defect injection, and parameter-specified editing) on single-part and assembly tracks. Alongside the code channel we provide a structured feature-op answer channel, in which the model emits an operation and its parameters and we apply verified primitives. Instruction-guided editing has two tiers, one that gives the defect's location and one in which the model must find it. Over five models, editing success in code spans up to 39 points, mostly through executability, and the structured channel narrows this to at most 8 in either tier. A model whose code runs on only 7% of instances when it must find the defect reaches frontier-level accuracy in the channel. The narrow band does not mean the models have converged. Many instances lie beyond what the channel can express and many are passed by every model, so fewer than 50 of the 301 single-part instances separate any two models in either tier, against at least 155 in code. Where the channel can express the repair, the two frontier models are indistinguishable when told the location and differ significantly when they must find it. Assemblies remain limited by the models, with wide spreads and no model near the channel's ceiling. We also report achievability oracles, a two-arm protocol run before any near-zero result is published. On our own pipeline they caught results that looked like model findings but were properties of the benchmark, including thirteen instances on which the metric rejects its own reference and 28 on which it accepts the unedited input.

Primary area:
datasets and benchmarks
Keywords:
CAD generation, CAD editting, domain specific language

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_q9TQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MCAD-Bench, a benchmark for evaluating CAD reconstruction and editing on real supplier-provided STEP models. It includes 144 industrial products and three task families: reconstruction, instruction-guided editing, and parameter-specified editing, covering both single parts and assemblies. The paper's main contribution is a controlled evaluation protocol that compares two answer channels: conventional CAD code generation and structured feature operations executed by a verified backend. It also introduces geometry-based editing metrics and achievability oracles to identify evaluation failures and measure the limitations of each answer channel. Experiments across multiple language models show that CAD API familiarity substantially affects code-based success rates, while structured operations improve executability but can also limit performance through their restricted expressiveness. The results highlight the importance of separating model capabilities from answer representation and evaluation artifacts.

Critical strengths
  1. Novel perspective on CAD benchmark evaluation. The paper highlights that CAD benchmark scores are influenced not only by geometric reasoning but also by the model's familiarity with specific CAD APIs. By comparing code-based and structured answer channels, the authors provide useful evidence of how output representation affects executability and model rankings. This offers an important methodological insight for designing more reliable CAD benchmarks.
  2. Careful evaluation design and validation. The benchmark uses real supplier CAD models and introduces localized editing metrics that assess both whether the intended modification is correct and whether unrelated geometry is preserved. The achievability oracles further identify invalid test cases and limitations of the structured channel. These validation efforts strengthen the credibility of the experimental findings and provide useful practices for future CAD benchmark development.
Critical weaknesses
  1. Limited comparability between the two answer channels. The structured channel not only changes the output format but also provides an explicit feature menu and a verified executor that handles feature matching and geometric operations. Although the authors include additional control experiments, the two channels still require different levels of effort from the models. This makes it difficult to fully isolate the effect of CAD API familiarity from the benefits of automated feature localization and execution.
  2. Limited representativeness of real-world CAD editing. Although the benchmark uses real supplier CAD models, its main editing tasks are generated by injecting a single predefined defect and asking the model to reverse it. This differs from practical CAD editing, which often involves multiple interdependent modifications and open-ended design requirements. As a result, it remains unclear how well the benchmark reflects the capabilities needed for real-world CAD editing workflows.
Questions for authors
  1. Answer channel comparability. The structured channel provides an explicit feature menu and a verified executor that performs feature matching and geometric operations, whereas the code channel requires models to handle these steps themselves. Although the paper includes a feature-menu control experiment, how much of the observed performance difference can be attributed specifically to removing CAD API requirements, rather than the additional capabilities provided by the structured executor?
  2. Benchmark representativeness. The instruction-guided editing tasks are constructed by injecting a single predefined defect into real supplier CAD models. To what extent do the authors believe these tasks reflect practical CAD editing capabilities? Could the authors provide evidence that the benchmark's conclusions remain valid for more complex or multi-step editing tasks?
3 comments

We thank the reviewer for a close reading. Both concerns are fair.

Q1. How much of the gap is the API, and how much is the executor? The structured channel changes three things at once. It gives the model a menu of the input's features, it asks for a short JSON answer instead of a program, and it adds an executor that finds the named feature and applies the edit. Three controls at the detection tier probe the menu, API knowledge and the executor in turn.

  1. Menu access (Table 2). We gave the code channel the same feature menu. More answers run, but success moves far less than through the structured channel. The open models gain 3 to 9 points with the menu, against 27 to 36 through the structured channel, and Claude loses 15. Access to pre-extracted coordinates is not the advantage.
  2. API reference. We gave the code channel a fixed API reference that names no instance's target. Table 15 covers Claude and GPT, and we have now run the three open models (rows marked new). The last column counts instances that only referenced code passes against those that only the structured channel passes, with the McNemar p.
modelcode+ referencestructuredpaired
Claude39%53%46%67 / 43 (p = .03)
GPT42%49%50%53 / 57 (p = .78)
GLM (new)15%40%44%46 / 57 (p = .32)
DS-Pro (new)3%28%45%40 / 92 (p = 7e-6)
DS-Flash (new)3%34%43%46 / 71 (p = .03)

The reference helps every model in code. It closes the gap to the structured channel for Claude, GPT and GLM. For the two weakest code writers the structured channel keeps a lead, clearly for DS-Pro, where more than half the answers still do not run, and marginally for DS-Flash.

The reference is not pure API knowledge, though. For GPT the gain is lexical. Without the reference, 59 of its 112 non-running answers call a method that does not exist. With it, more answers run, but its accuracy on those barely moves (66% to 64%). The other four models also become more accurate on answers that run (Claude 47% to 58%, the open models by 16 to 26 points), because the reference includes generic recipes for matching a feature by surface type and radius, which is the executor's job written in prose.

  1. Executor tolerance (Table 13). If the executor were rescuing imprecise coordinates, a strict tolerance would hurt. Requiring every stated coordinate to lie within 5% of the part diagonal of its true position costs none of the five models more than 1.6 points.

The executor also never locates the defect. The model chooses which feature to edit and with what parameters.

What these controls do not separate is the answer format from the executor. The clean test would expose the executor's operations as Python functions inside the code channel, and we have not run it. We agree the two channels ask different things of a model, which is why the paper recommends reporting both. The final version will name this open control, add the open-model rows to Table 15, and credit the advantage to the format and executor together, not to API removal alone.

Q2. Does single-defect editing represent real editing? A single injected defect is narrower than practical editing, which chains interdependent changes toward an open-ended goal. We chose it for measurement. Since we created the defect, we know its exact inverse, so every instance can be certified (the correct answer passes and doing nothing fails) and scored both on the edited region and on everything outside it. Our own audit shows why this matters. Before certification, the metric rejected its own reference on 13 instances and accepted the unedited input on 28, even with single defects. We see single-defect editing as a necessary first test, not a sufficient one. The best result is 63% (Fable, code channel, detection tier), so current models are not yet reliable even on one step.

We have no multi-step results and claim none. The closest evidence is on compound changes (new). The parametric track pairs real supplier variants, and 90 of its 217 pairs change several parameters at once. On these the channel effect keeps its direction but shrinks. On single-part targets the structured channel helps 6 of 7 models on multi-parameter pairs (mean mIoU 0.127 to 0.157), less than on single-parameter pairs (0.175 to 0.274). On multi-solid targets it raises exact solid count for 6 of 7 models on multi-parameter pairs (50% to 67%), against 49% to 79% on single-parameter ones. This track builds the target from its specification, so it tests compound changes, not multi-step editing.

Extending to several defects is direct. The scored region is the changed surface between the reference and the input, however many pieces it has, so a two-defect instance needs no metric change and is certified the same way. The final version will state this scope and include the by-kind results.

Thanks for your response and additional experiments. Most of my concerns have been addressed.

Q1: The new experiments explain the performance differences between the two channels. I appreciate the authors' acknowledgment that the effects of the output format and executor are not fully separated. I suggest making this limitation clear in the paper. Q2: I understand why chose single-defect editing tasks. The additional multi-parameter experiments are also helpful, but they do not directly demonstrate generalization to multi-step editing. I suggest mentioning this as a limitation in the paper.

Thank you for taking the time to review our responses and the additional experiments. We appreciate your constructive feedback, which has helped us improve the manuscript.

Reviewer_sicaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines an issue in benchmarking LLMs for CAD editing: existing benchmarks typically require models to generate executable CAD kernel code, conflating geometric reasoning ability with familiarity with a specific CAD API. To separate these factors, the paper introduces MCAD-Bench, a benchmark comprising 144 real supplier products in STEP format and three task families: reconstruction, instruction-guided editing through defect injection, and parameter-specified editing. The authors design two competing evaluation channels—a conventional code channel and a new structured feature-based channel—alongside achievability oracles. A key counterintuitive finding is that tightly clustered scores under the structured channel do not indicate converged model performance.

Critical strengths
  • This work addresses a critical evaluation issue in existing LLM‑for‑CAD benchmarks: prior evaluations conflate genuine geometric reasoning with the ability to write code for a specific CAD kernel API.
  • The paper built a MCAD-Bench from real supplier products stored as STEP. MCAD‑Bench leverages 144 real‑world supplier STEP parts and assemblies, rather than synthetic toy CSG geometries.
Critical weaknesses
  • The proposed structured feature‑op vocabulary is purpose‑built for localized repair‑style editing on pre‑defined feature types. There is limited discussion on how results from this controlled answer channel can generalize to broader CAD tasks.
  • The paper validates its central saturation claim only on a fixed set of 301 single-part instances using greedy sampling (temperature = 0). Experiments across difficulty-stratified subsets and with non-greedy sampling are needed to assess its robustness.
  • The writing is poor, making the paper difficult to read. Many domain-specific shorthand terms, such as “feature-op”, “sketch-op”, and “structured DSL,” are used without definition. The paper also overuses domain-specific or problem-specific terminology, such as channel, achievability oracle, and ceiling. Some sentences are especially difficult to understand, such as “The fix is not a better API, because there is no canonical one” and “Controlling the channel changes the panel.”
Questions for authors
  • For the structured feature channel, can you quantify which edit types it supports and what proportion of benchmark samples cannot be expressed in this DSL?
  • The saturation result is central to the paper and is based on temperature‑0 greedy sampling on 301 fixed single‑part instances. Does this pattern persist across difficulty-stratified easy, medium, and hard subsets, and under non-greedy sampling?
2 comments

We thank the reviewer for a clear summary and insightful views.

Presentation and terminology. The criticism is fair. "Feature-op" and "DSL" are used without definition, and other terms are introduced too tersely. The final version defines each at first use, as below.

  • Answer channel. The form in which a model returns its answer. In the code channel it writes CadQuery or OpenCascade code that we run. In the structured channel (the paper's "DSL") it returns one JSON operation, for example add_hole with a centre, axis and radius, chosen against a list of the input's own features, and our executor applies it.
  • Feature operation. An edit to one CAD feature, such as adding a hole or rounding an edge.
  • Panel. The set of evaluated models, which the final version calls simply "the models".
  • Achievability oracle. A check run before any result is reported. It submits the known correct answer, directly or through the channel, to test whether the instance can be passed at all.
  • Ceiling. The success rate when the correct answer is submitted through the structured channel.
  • Saturation. Below the ceiling, most instances are passed by every model or failed by every model, so few of them separate the models.

The two sentences the reviewer quotes become "Switching to a different kernel API would not solve this, because CAD systems share no standard API" and "When every model answers through the same structured channel, both the spread between models and their ranking change."

W1. Generalization. The edit vocabulary is narrow by design, since it inverts the injection operators, but the finding does not depend on it. Appendix G repeats the comparison for building a part from its specification, through a general construction vocabulary (sketch, extrude, revolve, loft, sweep, boolean, fillet, chamfer, shell, pattern) and a separate executor. All seven models improve (mean mIoU 0.139 to 0.200, executable 64% to 80%), and exact solid count on multi-solid targets rises from 52% to 75%. The channel stops working for re-dimensioning a supplier STEP, a global edit of a file with no feature history, where a closed vocabulary covers under a third of pairs (App. H). The final version will state this boundary.

Q1. Coverage. Single parts have seven operations (add, remove or resize a hole, add or remove a fillet or a chamfer), six of which have instances. Assemblies have three (move, duplicate or remove a solid). Of the 301 certified single-part instances, 18 (6%) cannot be expressed because the executor cannot resolve the target (16 blends, 2 bores). Another 99 fail even when the correct answer and up to eight alternatives are submitted, so the ceiling is 184 of 301 (61%). In code at least one model passes 34 of these 117 with the location given and 45 without, so they are hard for the channel, not impossible. On assemblies the oracle passes 120 of 126.

Q2. Difficulty (new). Share of the 301 instances that separate at least two of the five models, location given / withheld.

stratumstructuredcode
part complexity, face-count terciles7–9% / 13–18%46–55% / 62–65%
defect size, terciles6–11% / 12–21%47–57% / 57–66%
difficulty in code (0, 1–2, 3–5 pass)3–10% / 12–27%defines strata
remove a feature (72)18% / 38%42% / 44%
assemblies (126), for contrast61% / 36%67% / 38%

Saturation holds in every single-part stratum but one. When the model must find a removed feature, 38% of instances separate models, against 10% for add or resize. That is the localization effect of Sec. 5.1. The assembly row shows that the statistic does not report saturation by construction.

Q2. Sampling. We did not run a non-greedy arm, and every number in the paper is one greedy sample. We fixed temperature 0 because in an earlier revision two identical runs at the default temperature disagreed on about a fifth of instances (App. I). Whether saturation survives sampling is open. Measuring it properly needs per-instance pass rates over several samples in both channels, compared against a noise baseline, and the final version will name this as a limitation of the saturation result. The 117 instances beyond the ceiling are a separate matter. They fail even when the correct answer is submitted through the channel, and none of the five models passes any of them.

Thanks the authors for the response. The response partially addressed my concerns. I acknowledge that the key finding of the paper is interesting. However, I am still concerning about the overall quality of the paper, in terms of paper writing and experimental evaluation. I keep my original rating.

Reviewer_7cm7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces MCAD-Bench, which is a new benchmark for CAD models. It consists of a number of real STEP models, and covers several tasks - reconstruction, instruction-guided editing and some others. Authors study and compares how the model behaves, if the output is unrestricted code or structured feature operations. In most of models, the structures channel leads to better results, which can be explained by poor knowledge of specific API. But for strong models like Fable the results are reversed.

Critical strengths
  1. This paper studies an important factor of CAD AI models evaluation, whether specific API knowledge lead to errors in the execution. This is done by using 2 ouput channels - just unrestrictred code and structured output (DSL). In most of models, the structures channel leads to better results, which can be explained by poor knowledge of specific API. But for strong models like Fable the results are reversed.
  2. The proposed benchmark consists of a real-life complex objects, and can be useful.
Critical weaknesses

I think the key weakness is limited scope, novelty and impact of the work. The paper introduce a new benchmark, rather small, and there are a number of benchmark in the area. There is no new models or no novel approaches to solving the problem, just some studies on the proposed benchmarks in several tasks and modes. Overall it seems not significant enought for ICLR-level conference.

Questions for authors
  1. How the models in the benchmark have been selected? How can we be sure that is covers CAD topics well? Does it cover all CAD operations?
  2. Could you specify how exactly the results on your benchmark will advance the field of AI for CAD? What specific "take aways", which can be applied to other new methods are notable?
1 comment

We thank the reviewer for an accurate summary. The review raises no soundness concern, so we answer on significance, coverage and take-aways. Numbers marked new come from stored predictions and the dataset manifest, with no new model calls.

W1. Novelty. The contribution is a measurement result, not only a dataset. Most LLM benchmarks for CAD ask for a program in one kernel API, and we show that this choice determines much of the score. On the same 301 certified editing instances, five models span up to 39 points in code and at most 8 through a structured feature-operation channel (Fig. 1), and an API reference that names no target moves Claude from 39% to 53% (p = 2e-9, Table 15). P3D-Bench ranks output formats. We control the format and measure what that control costs.

Novel approaches. MCAD-Bench proposes no model, but four reusable evaluation methods. Pass@IoU and Pass@CD replace a single success line with a curve over thresholds and count a failed generation as failing at every threshold, so executability and accuracy enter one bounded score with no arbitrary penalty for the unbounded Chamfer distance (Sec. 4.2). A two-sided editing metric rewards the requested change and penalizes collateral damage above a measured noise floor. A controlled answer channel with a calibrated executor makes the format effect measurable. Achievability oracles removed 41 of the 342 generated single-part instances (12%) that our own metric could not score correctly, which would otherwise have counted as model results.

Size. MCAD-Bench is test-only, with 427 certified editing instances and 85 reconstruction targets from 144 supplier products of up to 4019 faces and 83 solids, and 217 parameter pairs from 149 more. That is the scale of the concurrent benchmarks with assemblies, P3D-Bench (300 evaluated cases) and ArtiCAD-Bench (120 tasks). BenchCAD (17,900 generated parts, also used for training) and CADBench (18,000 shapes resampled from public corpora) are larger but contain no assemblies.

Q1. Selection and coverage. We sampled 150 products from 3DSource, a library of supplier-authored CAD, across its taxonomy rather than by catalogue volume, so each of its 74 component types contributes a median of two products and at most five (new). Six were excluded for unreadable or non-millimetre STEP. The 144 span nine categories from fasteners to motors and sizes from 4.4 mm to 3.0 m. MCAD-Bench covers mechanical components, not all of CAD. Its 51,310 faces are 45% planar, 27% cylindrical, 10% conical, 6% toroidal and 10% B-spline (new), and freeform styling is out of scope. If the question concerns the evaluated LLMs, they are two frontier and three open-weight models on one gateway at temperature 0 (Sec. 5).

All operations? No. Editing has six single-part operators (add, remove or resize a hole, add a fillet or chamfer, remove a fillet) and three assembly ones (displace a component, break a contact, delete a member of a symmetry group). Each needs an exact inverse we can certify, which the two-sided metric and the oracles require, and the single-part ones target the cylindrical, conical and toroidal faces that make up 43% of the corpus. Shelling, drafting and patterning are absent. Parameter changes form a separate, globally scored family, and reconstruction is open-vocabulary.

Robustness by category (new). In all six categories with at least 14 instances, the spread across models at detection falls from 34–51 points in code to 3–15 in the structured channel.

Q2. Take-aways

  1. Report executability and an API-reference control with every code-channel score. Without the reference, 59 of GPT's 112 non-executing answers call a method that does not exist.
  2. Measure a tool or DSL interface's ceiling before adopting it. Ours lifts weak models by up to 42 points and costs the strongest 13.
  3. Report the separating subset of a flat panel. Only 25 to 49 of 301 instances separate any two models in our structured channel, against 155 to 190 in code.
  4. Work on localization. The frontier models are indistinguishable when told where the defect is and separated when they must find it, and feature removal is hardest.
  5. Run degenerate baselines and achievability oracles. Doing nothing passed 28 instances before certification, and same-product retrieval beats all seven models on specification-conditioned generation (Table 3). The tools ship with the benchmark.

We will fold these clarifications, the take-aways and the comparison with concurrent benchmarks into the final version.

Comments on the paper

1 comment

We thank the three reviewers. All three recognize the problem the paper addresses (CAD benchmark scores mix geometric reasoning with fluency in one kernel API) and the value of building on real supplier STEP. This note collects the new analyses from the three threads.

New analyses

  • API reference for all five models (q9TQ, new runs). The paper ran this control for the two frontier models only. Run for the three open models, it lifts code-channel success from 15% to 40% for GLM, 3% to 28% for DS-Pro and 3% to 34% for DS-Flash. The reference closes the gap to the structured channel for Claude, GPT and GLM. The structured channel keeps a lead for the two DeepSeek models, clearly for DS-Pro and marginally for DS-Flash. The reference also carries generic feature-matching recipes, so it is not pure API knowledge.
  • Saturation by difficulty (sica). On the same 301 instances, the share that separates any two models through the structured channel stays at 6–11% with the location given and 12–21% when the model must find it, in every tercile of part complexity and of defect size, against 46–57% and 57–66% in code. The one exception is feature removal when the model must find the defect (38%), which is the localization effect the paper reports.
  • Channel effect by product category (7cm7). At the detection tier, the five models' spread falls from 34–51 points in code to 3–15 in the structured channel in all six categories with at least 14 instances.
  • Channel effect by kind of parameter change (sica, q9TQ). In the parametric track the structured channel helps 6 of 7 models on single-parameter and on multi-parameter changes, by less on the latter.
  • Corpus census (7cm7). The 144 products hold 51,310 faces, 45% planar, 27% cylindrical, 10% conical, 6% toroidal, 1% spherical and 10% B-spline.

Revision. We will fold the clarifications made in each thread into the final version, among them the new analyses above, definitions of our terms, the scope of the benchmark and of the structured channel, the attribution of the channel effect, a sampling caveat on the saturation result, a comparison with concurrent benchmarks, and take-aways for method developers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.