- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper tackles the problem of functional validity of AI generated CAD models with AI agents. While recent AI models/agents showed impressive performance for the generation of CAD models, it is unclear whether their functional usage is valid. This paper proposes (1) a harness that turns funtional requirements into simulation tests assessing AI generated CAD model functional validity, while using them as feedback signals to revise the AI generated CAD models; (2) a benchmark of 46 different mechanism categories, with input functional requirements and the corresponding simulation tests assessung frontier agents and some other baselines. The experiments show that all tested AI agents and baselines cannot pass sucessfully all the tests and integrating feedback from the tests helps in improving their performance.
Critical strengths
-
The main strength of this paper is the scope of the study. Recent AI agents/models are getting better and better at generating visually plausible parts from different inputs (text, images, point clouds, etc.), with many works showcasing increasingly better capabilities for this task. However, functional validity remains largely overlooked. When designers/engineers create CAD parts, they create them to achieve a certain function. AI models for CAD generation should do so, which is the natural next step to take. However, there is a lack in litterature on studies trying to evaluate them for functionality, or to make them better at this task. This paper presents one of the first attempts to investigate this and can pave the way for further attempts in the field to focus on this very important yet overlooked problem.
-
The paper not only showcases results on functional CAD design but also shows that the simulation tests improve CAD generation for functionality.
Critical weaknesses
- The main weakness of this paper consists of the generated simulation tests which themselves seem to be generated by AI agents. The authors already mentioned in discussion section issues related to functional test coverage which is a common problem for test-based evaluation of software generation. If these tests are generated by AI agents, how can we guarantee their reliability as a metric for the benchmark or as a signal for improving CAD generation? At least a study on how reliable are the considered tests are should have been conducted. The authors could have looked at the validity of the tests by running them on ground-truth CAD models (if existing), or looking at coverage metrics based on mutations like the one considered in [X]. A more in-depth study on how these tests were constructed and a quantitative evaluation of their reliablity as a metric for functional CAD design is needed.
[X] Mallis, D., Wang, M., Karadeniz, A. S., Ricci, E., Kacem, A., & Aouada, D. (2026). Text-to-CAD Evaluation with CADTests. arXiv preprint arXiv:2605.07807.
- Several aspects related to the dataset preparation seem to be unclear and lack real-world grounding, particularly the selection of CAD models, the definition of functional requirements, and the categorization of different mechanisms. The paper would benefit from an explanation grounded by CAD expertise, of how these design choices reflect real-world functional constraints and representative engineering scenarios. For example, the authors could have conducted a human expert study showcasing the validity of the functional requirements and their form and assessing the alignment of the conducted evaluation with human experts (e.g., CAD designers).
Questions for authors
-
In which form the considered AI agents generate CAD models? Python CAD programs (e.g., CadQuery) or BReps or any other form?
-
How the functional tests were created? Are these created by AI agents? In this case, how can one ensure that these functional tests are valid and covering all necessary aspects of the functional requirements?
-
How is intended use, called as functional requirements, is formulated here? how this is close to functional requirements used in real design workflows?
-
How were the 46 families constructed? Have these been provided by CAD experts? Are there ground-truth CAD models constructed by human designers for these familiies?
4 comments
Response to Reviewer 4hcy (2/2)
The paper would benefit from an explanation grounded by CAD expertise ... For example, the authors could have conducted a human expert study showcasing the validity of the functional requirements and their form and assessing the alignment of the conducted evaluation with human experts.
As suggested, 17 experts with more than two years of CAD experience took part in a human study on 10 representative tasks. Each expert read the requirements and tests of five tasks and answered four yes/no questions. Each also judged six designs from other tasks, half passed and half failed by the harness, after watching the simulation replay.
| Question | Yes |
|---|---|
| The requirements are reasonable | 85/85 |
| They cover the main functions of the mechanism | 75/85 |
| The test is reasonable | 83/85 |
| The test checks all the requirements | 81/85 |
| A design the harness passes meets the requirements | 46/51 |
On designs the harness fails, experts mostly judged the motion to look right, since violations such as a slight interpenetration do not show in an animation. The harness catches them from the recorded trajectories (see our reply to Reviewer 1FEV). We have added the expert study to the revised paper.
In which form the considered AI agents generate CAD models? Python CAD programs (e.g., CadQuery) or BReps or any other form?
The agents write Python programs with build123d, a CAD library on the same OpenCASCADE kernel as CadQuery, which build B-Rep solids. Each design is submitted as the Python source, a multi-body STEP assembly and its physical declarations, i.e., rigid bodies, joints, actuators and materials (Appendix A.1). Project Chrono simulates collision meshes derived from the B-Rep, and the geometry checks run on the B-Rep itself (Appendix D.3). We have named the library in Section 3.1 of the revised paper.
How is intended use, called as functional requirements, is formulated here? how this is close to functional requirements used in real design workflows?
Each task starts with a short natural-language request, followed by a list of functional requirements. A requirement states an observable behavior under a given input or load, often with a target value and, where needed, a dependence on earlier states. The task also states fixed constraints and permitted operations, and leaves the parts and architecture open (Appendix A.1). For example, the winch (Task 42) must lift a load through a reduction, hold it against a reverse load, and release it and lock again.
This matches how requirements are written in engineering design. Before concepts are generated, a design specification lists solution-neutral requirements with measurable targets, as in the requirements list of systematic design [1] and the functional requirements of axiomatic design [2]. We have added this explanation to Section 3 of the revised paper.
[1] G. Pahl, W. Beitz, J. Feldhusen, and K.-H. Grote. Engineering Design: A Systematic Approach. Springer, 2007.
[2] N. P. Suh. The Principles of Design. Oxford University Press, 1990.
Response to the authors’ rebuttal
I thank the authors for their detailed response and the substantial additional work. The mutation study gives the quantitative check of test reliability that was missing, and detecting 39 of 40 faults with no false rejections is convincing. The expert study with 17 CAD practitioners is also a welcome addition, and it shows that the requirements and tests are reasonable and aligned with expert judgment. The clarifications on dataset construction, the requirement formulation grounded in systematic and axiomatic design, the CAD representation and the reference designs for all 46 tasks address my remaining concerns.
One clarification question: were the mutations introduced manually by the authors or generated by an agent? Please state this in the revised paper.
We sincerely thank the reviewer for the positive assessment and for engaging with our response.
Regarding the follow-up question, the mutations were generated by a coding agent under fault classes and rules that we fixed in advance, and we manually checked every mutation. The rules are a single change at one location, numeric shortfalls of 20–40%, and unchanged design notes. For each mutation, we inspected the file diff, the renders, a before/after 3D view and the corresponding simulation results to confirm that the intended change is the only one and that it breaks the stated function. All mutations were frozen before the harness was run. We have added these details to Section 5.1 and Appendix H.3 of the revised paper.
Response to Reviewer 4hcy (1/2)
We appreciate the reviewer's time and efforts in reviewing our paper and their detailed and constructive comments.
For the 46 benchmark tasks, the test templates were first drafted with the help of an LLM, then revised and checked by the authors to make sure every functional requirement in Table 4 is tested. We now state this in Appendix C.1. The same template is used for all designs of a task (Section 4.1). Agents write templates only for new open-ended requests, and these templates are reviewed by another agent (Appendix A.5).
An agent is used for test binding. The same requirement can be met by different mechanisms, e.g., different gear trains or latch shapes, so a test cannot refer to fixed part names. For each design, a binding agent (not the constructor) connects the template's actuators, loads and measured signals to the parts of that design (Appendix A.3). Whether the design passes is then computed by the simulation code from the recorded trajectories. When we let other LLMs do the binding, the results are nearly the same (see our reply to Reviewer 1FEV). We check the validity and coverage of the tests in the mutation and expert studies below.
We thank the reviewer for pointing out this important issue. We agree that a quantitative check was missing, and we have run a mutation study along the lines the reviewer suggests.
A reliable test should reject a design with a fault and accept the same design without it. Our tasks have no single ground-truth CAD model, since each can be met by different mechanisms. Following CADTests [X], we therefore took 40 passing designs from 23 tasks (10 per constructor) and introduced one fault into each. Each mutated design was tested with the same template, and the binding agent bound it again without being told about the fault. As a control, we also re-tested 10 of the original designs unchanged.
The tests caught 39 of the 40 faults and rejected none of the unchanged designs. The faults were caught by pre-simulation checks, geometry checks, functional simulation or the simulation time limit. The one miss is a door stop placed on a body that the simulator excludes from collision, so the stop had no effect.
In our expert study below, experts also judged that the tests check all the requirements in 81 of 85 answers. We have added the mutation study to the revised paper.
[X] D. Mallis, M. Wang, A. S. Karadeniz, E. Ricci, A. Kacem, and D. Aouada. Text-to-CAD Evaluation with CADTests. arXiv preprint arXiv:2605.07807, 2026.
FunCADBench provides no CAD models. Each task states the functional requirements of a mechanism, and the agent builds the CAD model. Public CAD models (e.g., on Printables and GitHub) exist for many families but are made for 3D printing to other specifications. For example, public watch models have no alarm.
The 46 families come from off-the-shelf products, mainly in the McMaster-Carr catalog, and from everyday mechanisms. Their requirements follow what each product is used for, e.g., a winch must lift, hold and safely release a load (see also our reply on functional requirements below). The families span five axes of mechanical demand: motion primitives, topology, complexity, contact and geometric difficulty (Section 3.2, Table 1).
Every task has a design that passes its tests. Agents passed 40 tasks across our runs, including the Grok + history setting (Section 5.1), whose per-task results we have added to the revised paper. For five of the other six (Tasks 7, 9, 20, 45 and 46), a human completed the closest failed agent design. For Task 34, we adapted an openly licensed public ratchet design. All six pass the unchanged tests. We will release these designs with the benchmark, and we have added this description of the dataset to Section 3 and Appendix H.1 of the revised paper.