- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MCAD-Bench, a benchmark for evaluating CAD reconstruction and editing on real supplier-provided STEP models. It includes 144 industrial products and three task families: reconstruction, instruction-guided editing, and parameter-specified editing, covering both single parts and assemblies. The paper's main contribution is a controlled evaluation protocol that compares two answer channels: conventional CAD code generation and structured feature operations executed by a verified backend. It also introduces geometry-based editing metrics and achievability oracles to identify evaluation failures and measure the limitations of each answer channel. Experiments across multiple language models show that CAD API familiarity substantially affects code-based success rates, while structured operations improve executability but can also limit performance through their restricted expressiveness. The results highlight the importance of separating model capabilities from answer representation and evaluation artifacts.
Critical strengths
- Novel perspective on CAD benchmark evaluation. The paper highlights that CAD benchmark scores are influenced not only by geometric reasoning but also by the model's familiarity with specific CAD APIs. By comparing code-based and structured answer channels, the authors provide useful evidence of how output representation affects executability and model rankings. This offers an important methodological insight for designing more reliable CAD benchmarks.
- Careful evaluation design and validation. The benchmark uses real supplier CAD models and introduces localized editing metrics that assess both whether the intended modification is correct and whether unrelated geometry is preserved. The achievability oracles further identify invalid test cases and limitations of the structured channel. These validation efforts strengthen the credibility of the experimental findings and provide useful practices for future CAD benchmark development.
Critical weaknesses
- Limited comparability between the two answer channels. The structured channel not only changes the output format but also provides an explicit feature menu and a verified executor that handles feature matching and geometric operations. Although the authors include additional control experiments, the two channels still require different levels of effort from the models. This makes it difficult to fully isolate the effect of CAD API familiarity from the benefits of automated feature localization and execution.
- Limited representativeness of real-world CAD editing. Although the benchmark uses real supplier CAD models, its main editing tasks are generated by injecting a single predefined defect and asking the model to reverse it. This differs from practical CAD editing, which often involves multiple interdependent modifications and open-ended design requirements. As a result, it remains unclear how well the benchmark reflects the capabilities needed for real-world CAD editing workflows.
Questions for authors
- Answer channel comparability. The structured channel provides an explicit feature menu and a verified executor that performs feature matching and geometric operations, whereas the code channel requires models to handle these steps themselves. Although the paper includes a feature-menu control experiment, how much of the observed performance difference can be attributed specifically to removing CAD API requirements, rather than the additional capabilities provided by the structured executor?
- Benchmark representativeness. The instruction-guided editing tasks are constructed by injecting a single predefined defect into real supplier CAD models. To what extent do the authors believe these tasks reflect practical CAD editing capabilities? Could the authors provide evidence that the benchmark's conclusions remain valid for more complex or multi-step editing tasks?
3 comments
Thanks for your response and additional experiments. Most of my concerns have been addressed.
Q1: The new experiments explain the performance differences between the two channels. I appreciate the authors' acknowledgment that the effects of the output format and executor are not fully separated. I suggest making this limitation clear in the paper. Q2: I understand why chose single-defect editing tasks. The additional multi-parameter experiments are also helpful, but they do not directly demonstrate generalization to multi-step editing. I suggest mentioning this as a limitation in the paper.
Thank you for taking the time to review our responses and the additional experiments. We appreciate your constructive feedback, which has helped us improve the manuscript.
We thank the reviewer for a close reading. Both concerns are fair.
Q1. How much of the gap is the API, and how much is the executor? The structured channel changes three things at once. It gives the model a menu of the input's features, it asks for a short JSON answer instead of a program, and it adds an executor that finds the named feature and applies the edit. Three controls at the detection tier probe the menu, API knowledge and the executor in turn.
The reference helps every model in code. It closes the gap to the structured channel for Claude, GPT and GLM. For the two weakest code writers the structured channel keeps a lead, clearly for DS-Pro, where more than half the answers still do not run, and marginally for DS-Flash.
The reference is not pure API knowledge, though. For GPT the gain is lexical. Without the reference, 59 of its 112 non-running answers call a method that does not exist. With it, more answers run, but its accuracy on those barely moves (66% to 64%). The other four models also become more accurate on answers that run (Claude 47% to 58%, the open models by 16 to 26 points), because the reference includes generic recipes for matching a feature by surface type and radius, which is the executor's job written in prose.
The executor also never locates the defect. The model chooses which feature to edit and with what parameters.
What these controls do not separate is the answer format from the executor. The clean test would expose the executor's operations as Python functions inside the code channel, and we have not run it. We agree the two channels ask different things of a model, which is why the paper recommends reporting both. The final version will name this open control, add the open-model rows to Table 15, and credit the advantage to the format and executor together, not to API removal alone.
Q2. Does single-defect editing represent real editing? A single injected defect is narrower than practical editing, which chains interdependent changes toward an open-ended goal. We chose it for measurement. Since we created the defect, we know its exact inverse, so every instance can be certified (the correct answer passes and doing nothing fails) and scored both on the edited region and on everything outside it. Our own audit shows why this matters. Before certification, the metric rejected its own reference on 13 instances and accepted the unedited input on 28, even with single defects. We see single-defect editing as a necessary first test, not a sufficient one. The best result is 63% (Fable, code channel, detection tier), so current models are not yet reliable even on one step.
We have no multi-step results and claim none. The closest evidence is on compound changes (new). The parametric track pairs real supplier variants, and 90 of its 217 pairs change several parameters at once. On these the channel effect keeps its direction but shrinks. On single-part targets the structured channel helps 6 of 7 models on multi-parameter pairs (mean mIoU 0.127 to 0.157), less than on single-parameter pairs (0.175 to 0.274). On multi-solid targets it raises exact solid count for 6 of 7 models on multi-parameter pairs (50% to 67%), against 49% to 79% on single-parameter ones. This track builds the target from its specification, so it tests compound changes, not multi-step editing.
Extending to several defects is direct. The scored region is the changed surface between the reference and the input, however many pieces it has, so a two-defect instance needs no metric change and is certified the same way. The final version will state this scope and include the by-kind results.