- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents a benchmark of industrial parts and systems for CAD. It includes 1400 text descriptions of these parts intended for examining the ability of generative models to create 3D geometries that follow the text descriptions. The benchmark includes 40 categories and evaluates two tasks: category based generation and editing. The authors offer evaluation metrics and evaluates well known foundation models on these generation tasks.
Critical strengths
The benchmark and metrics presented are clear and can be useful for evaluating the ability of generative models to generate reliable designs and edit them in various industrial categories. The descriptions in the supplementary material seem detailed. I found the paper easy to follow and the benchmark seems to have good potential in my view.
As I am not very familiar with the field its hard for me to assess how big of a contribution of such benchmark is, and how much it is actually needed. The paper does provide a comparison to a previous benchmark which seems much less robust. As this is a benchmark paper technical correctness does not seem to be an issue and the proposed evaluation metrics seems reasonable to me. For these reasons, and assuming that there is an actual need for such benchmarks I was left with a positive impression.
Critical weaknesses
As this is a benchmark paper I did not find any critical weaknesses.
More explanation on how the examples were gathered and why (motivation, complexity, materials, etc.) is needed.
Adding more models to the presented evaluation could prove useful for future work.
Questions for authors
Can you explain the rationale of the selection and choices of the models in the benchmark.
Can you give a measure for the complexity of CAD models in the benchmark.
i appreciate the rebuttal and thank you for agreeing to make the results consistent. i have raised my score.