- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RealCADBench, a benchmark for evaluating parametric CAD modeling from real industrial design intents. It contains 12,632 tasks across 19 factory-automation categories, covering text descriptions, 2D engineering drawings, real product pictures, and rendered images for both part and assembly modeling. The authors propose an evaluation protocol combining executability, Solid IoU, Surface IoU, and a visual-semantic Judge to assess different aspects of CAD generation quality. Experiments on 1,770 tasks with nine frontier models and two coding agents show that strong executability does not necessarily lead to accurate geometry or semantic fidelity, and that model performance varies across input modalities and evaluation metrics.
Critical strengths
- Real-world industrial CAD benchmark. RealCADBench covers 12,632 tasks across 19 industrial product categories, with diverse inputs including text, engineering drawings, real product pictures, and rendered images. This provides a valuable testbed for evaluating CAD generation models beyond synthetic or CAD-native datasets.
- Multi-dimensional evaluation. The benchmark evaluates executability, geometric accuracy, and visual-semantic identity separately. This is useful because a valid CAD program does not necessarily produce accurate geometry or preserve important product features. The experiments also provide insights into how different models and agents perform across these dimensions.
Critical weaknesses
- Concerns about ground-truth reliability. For most non-rendered tasks, the reference 3D models are generated using AI-based reconstruction methods rather than original industrial CAD files. Although the authors apply expert verification, the reconstructed references may contain geometric inaccuracies or assumptions about invisible structures. This raises concerns about the reliability of IoU-based evaluation.
- Limited evaluation of parametric CAD modeling. Although the benchmark focuses on parametric CAD modeling, the evaluation only considers the exported STL geometry and does not assess feature histories, parametric constraints, or editability. As a result, the benchmark cannot distinguish a well-structured parametric CAD model from a geometrically similar but non-parametric result, limiting its ability to evaluate the claimed task.
Questions for authors
- Ground-truth reliability. For most non-rendered tasks, the reference CAD models are reconstructed using AI-based methods rather than original industrial CAD files. How do the authors ensure that these references are geometrically accurate, especially for hidden or ambiguous structures? Could the authors provide quantitative evidence of their reliability beyond expert screening?
- Parametric CAD evaluation. The benchmark evaluates exported STL geometry without directly assessing feature histories, parametric constraints, or editability. Could the authors clarify why these properties are excluded, given the benchmark's focus on parametric CAD modeling?
- Judge reliability. The paper validates the visual-semantic Judge mainly through cross-model consistency on six systems. Could the authors provide a direct comparison with independent human expert ratings to establish whether the Judge reliably reflects CAD modeling quality?
3 comments
Thank you for the response and clarifications. I appreciate the authors' acknowledgment of the current limitations. However, my main concerns about ground-truth reliability and Judge validation remain unresolved. I will maintain my current rate.
I think we have largely resolved your main concerns (Let me explain).
First, regarding the reliability of the Judge Score, we conducted a correlation analysis between human expert ratings and Judge Scores. Human ratings were provided on a 0–10 scale, and we discretized the Judge Scores into ten bins for comparison. The resulting Pearson correlation coefficient was approximately 0.71, indicating a positive association between the two assessments, although agreement was not perfect. We also qualitatively compared multiple outputs generated by the same model for identical inputs. In these comparisons, the Judge Score was sensitive to visible structural degradation, assigning lower scores to outputs with more pronounced structural errors.
Second, regarding reference-shape reliability, references outside the rendered-image subset were also assessed by human experts using a dedicated annotation tool. This assessment considered both external geometry and internal structure, where supported by the available input evidence. Image-to-3D models can draw on learned object priors to propose plausible internal structures, and expert assessment helps identify candidates consistent with the inputs. This is particularly relevant to the engineering-drawing subset, where dimensions, sectional views, or other structural details, when present, provide additional constraints on geometry that may not be visible in an ordinary photograph.
We will revise the manuscript to incorporate these clarifications and make the supporting evidence explicit. Specifically, we will describe the human–Judge correlation analysis, including the rating scales, score transformation, sample size, and evaluation protocol, and report the observed Pearson correlation alongside representative qualitative comparisons. We will also expand the reference-construction section to explain the annotation tool, the expert assessment criteria, and how the available input evidence is used to assess external geometry and internal structure. These revisions will distinguish directly verifiable geometric features from plausible but unverified completions, making both the evidence and its scope clearer to readers.
Response to Reviewer T7ch
Thank you for the review. We address reference accuracy, parametric evaluation, and Judge validation separately, using the evidence available in the manuscript.
1. Reference provenance and existing quantitative results.
We maintain that the two reference-construction routes must be assessed separately.
For 373 rendered-image tasks, both the input and reference originate from the same verified STEP asset. Non-rendered tasks use candidate reconstructions generated with Hunyuan-3D, followed by screening and practitioner cross-checks. The selection criteria adapt the Part Judge rubric to a 0-10 scale. These procedures establish the stated provenance and curation process; they do not independently quantify error in hidden or reconstructed geometry.
The appendix separately reports results for the CAD-derived regime. Across six frontier-scale models, mean executability is 0.8123, Solid IoU 0.5379, and Surface IoU 0.2165. These are measurements of model outputs against CAD-derived references, not measurements of the reconstruction-reference pipeline's accuracy. A withheld-CAD audit of that pipeline is not reported.
The Judge compares generated-output renders with input evidence without receiving the reference mesh; text tasks use the corresponding product photograph. Its information differs from that used by IoU, but this does not make it independent ground truth or a test of invisible geometry. Non-rendered IoU should be interpreted as agreement with curated reconstructed references.
2. Geometric reconstruction and parametric quality.
We stand by the geometric measurements within their stated scope.
The required output is an executable FreeCAD program, and the scoring artifact is its exported geometry. Solid IoU measures occupied volume and Surface IoU measures boundaries. Both respond to the geometric consequences of errors in relative dimensions, feature proportions, and spatial structure. Judge Score assesses visible shape, salient features, identity, and functional or assembly plausibility where applicable.
These measurements support comparisons of execution and realized geometry. They do not measure feature-history quality, constraint structure, or successful recomputation after edits. Uniform-scale normalization also removes absolute dimensions. The concern about parametric-model quality therefore identifies an unmeasured capability; we will explicitly restrict the claims to the capabilities assessed by the current protocol. No conclusion about editability or manufacturing readiness follows from STL overlap alone.
3. What the Judge consistency experiment establishes.
We stand by the six-system results as evidence of automated consistency.
Section A.5 reports a consistency check on six BenchCAD systems, separate from RealCADBench construction and prompt development. Two Kimi K2.6 runs differ by 0.90 points on average and preserve 13/15 pairwise system orderings. Each agrees with GPT-5.5 on 14/15 pairs and selects the same top-ranked system.
This experiment uses no human ratings. It cannot answer the request for independent human-Judge agreement, and agreement between automated judges does not rule out shared bias. Expert participation in prompt development and post-checks likewise differs from blinded evaluation on held-out outputs. We do not report a complete independent human-validation study here.
The supported use of the consistency result is to characterize one aspect of the frozen Judge's behavior. Judge scores remain separately reported alongside executability and geometric metrics, with their validation scope stated. A direct human comparison requires documented sample selection, rating protocols, independence from prompt development, and paired scores.
4. Public resource and scope of contribution.
We maintain that the publicly released industrial task resource is a concrete contribution.
RealCADBench-V1.0 is publicly accessible, with inputs, reference assets, and metadata available for inspection. The resource covers industrial text, drawings, photographs, and renders, with part and assembly tasks under a common executable-output protocol. Its existing contribution is this task resource and the separate measurement of execution, geometry, and visual-semantic quality. The reference-audit and human-validation questions remain distinct from the availability of that resource.