acceptodds
Under review as a conference paper at ICLR 2027

Can It Actually Work? Synthesizing Functional CAD Mechanisms

14 views

Abstract

LLM agents can construct sophisticated CAD models, but visually plausible assemblies need not fulfill their intended functions. We study functional CAD design, in which agents synthesize and revise CAD mechanisms to satisfy functional requirements in physics simulation. The same requirement can be realized by different mechanisms, so testing must account for each candidate’s controls, components, and operating sequence. To study this problem, we introduce FunCAD-Harness , a harness that connects functional requirements to candidate-specific simulation tests and uses their execution results to guide design revision. We also introduce FunCADBench, spanning 46 mechanism families with dependent and repeated operations, and evaluate frontier agents and baseline workflows. Our experiments show that simulation-guided revision through our harness exposes mechanical failures that are difficult to detect visually and improves functional task success. We further find that strong visual 3D modeling capabilities do not necessarily translate into superior functional design, with less expensive models achieving competitive performance at lower API cost. Together, our benchmark and harness take a step toward advancing LLM agents from geometric modeling to the design of mechanisms that address practical needs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM agents, functional CAD generation, mechanical design, functional testbed synthesis, physical reasoning, agent evaluation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_4hcyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper tackles the problem of functional validity of AI generated CAD models with AI agents. While recent AI models/agents showed impressive performance for the generation of CAD models, it is unclear whether their functional usage is valid. This paper proposes (1) a harness that turns funtional requirements into simulation tests assessing AI generated CAD model functional validity, while using them as feedback signals to revise the AI generated CAD models; (2) a benchmark of 46 different mechanism categories, with input functional requirements and the corresponding simulation tests assessung frontier agents and some other baselines. The experiments show that all tested AI agents and baselines cannot pass sucessfully all the tests and integrating feedback from the tests helps in improving their performance.

Critical strengths
  • The main strength of this paper is the scope of the study. Recent AI agents/models are getting better and better at generating visually plausible parts from different inputs (text, images, point clouds, etc.), with many works showcasing increasingly better capabilities for this task. However, functional validity remains largely overlooked. When designers/engineers create CAD parts, they create them to achieve a certain function. AI models for CAD generation should do so, which is the natural next step to take. However, there is a lack in litterature on studies trying to evaluate them for functionality, or to make them better at this task. This paper presents one of the first attempts to investigate this and can pave the way for further attempts in the field to focus on this very important yet overlooked problem.

  • The paper not only showcases results on functional CAD design but also shows that the simulation tests improve CAD generation for functionality.

Critical weaknesses
  • The main weakness of this paper consists of the generated simulation tests which themselves seem to be generated by AI agents. The authors already mentioned in discussion section issues related to functional test coverage which is a common problem for test-based evaluation of software generation. If these tests are generated by AI agents, how can we guarantee their reliability as a metric for the benchmark or as a signal for improving CAD generation? At least a study on how reliable are the considered tests are should have been conducted. The authors could have looked at the validity of the tests by running them on ground-truth CAD models (if existing), or looking at coverage metrics based on mutations like the one considered in [X]. A more in-depth study on how these tests were constructed and a quantitative evaluation of their reliablity as a metric for functional CAD design is needed.

[X] Mallis, D., Wang, M., Karadeniz, A. S., Ricci, E., Kacem, A., & Aouada, D. (2026). Text-to-CAD Evaluation with CADTests. arXiv preprint arXiv:2605.07807.

  • Several aspects related to the dataset preparation seem to be unclear and lack real-world grounding, particularly the selection of CAD models, the definition of functional requirements, and the categorization of different mechanisms. The paper would benefit from an explanation grounded by CAD expertise, of how these design choices reflect real-world functional constraints and representative engineering scenarios. For example, the authors could have conducted a human expert study showcasing the validity of the functional requirements and their form and assessing the alignment of the conducted evaluation with human experts (e.g., CAD designers).
Questions for authors
  • In which form the considered AI agents generate CAD models? Python CAD programs (e.g., CadQuery) or BReps or any other form?

  • How the functional tests were created? Are these created by AI agents? In this case, how can one ensure that these functional tests are valid and covering all necessary aspects of the functional requirements?

  • How is intended use, called as functional requirements, is formulated here? how this is close to functional requirements used in real design workflows?

  • How were the 46 families constructed? Have these been provided by CAD experts? Are there ground-truth CAD models constructed by human designers for these familiies?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4hcy (1/2)

We appreciate the reviewer's time and efforts in reviewing our paper and their detailed and constructive comments.

How the functional tests were created? Are these created by AI agents? In this case, how can one ensure that these functional tests are valid and covering all necessary aspects of the functional requirements?

For the 46 benchmark tasks, the test templates were first drafted with the help of an LLM, then revised and checked by the authors to make sure every functional requirement in Table 4 is tested. We now state this in Appendix C.1. The same template is used for all designs of a task (Section 4.1). Agents write templates only for new open-ended requests, and these templates are reviewed by another agent (Appendix A.5).

An agent is used for test binding. The same requirement can be met by different mechanisms, e.g., different gear trains or latch shapes, so a test cannot refer to fixed part names. For each design, a binding agent (not the constructor) connects the template's actuators, loads and measured signals to the parts of that design (Appendix A.3). Whether the design passes is then computed by the simulation code from the recorded trajectories. When we let other LLMs do the binding, the results are nearly the same (see our reply to Reviewer 1FEV). We check the validity and coverage of the tests in the mutation and expert studies below.

At least a study on how reliable are the considered tests are should have been conducted. The authors could have looked at the validity of the tests by running them on ground-truth CAD models (if existing), or looking at coverage metrics based on mutations like the one considered in [X].

We thank the reviewer for pointing out this important issue. We agree that a quantitative check was missing, and we have run a mutation study along the lines the reviewer suggests.

A reliable test should reject a design with a fault and accept the same design without it. Our tasks have no single ground-truth CAD model, since each can be met by different mechanisms. Following CADTests [X], we therefore took 40 passing designs from 23 tasks (10 per constructor) and introduced one fault into each. Each mutated design was tested with the same template, and the binding agent bound it again without being told about the fault. As a control, we also re-tested 10 of the original designs unchanged.

Fault typeMutated designsDetected
Cannot be simulated88
Interpenetration88
Wrong mechanical principle (e.g., reversed spring or latch bevel)87
Wrong connection or constraint88
Numeric target missed by 20-40%88
All4039

The tests caught 39 of the 40 faults and rejected none of the unchanged designs. The faults were caught by pre-simulation checks, geometry checks, functional simulation or the simulation time limit. The one miss is a door stop placed on a body that the simulator excludes from collision, so the stop had no effect.

In our expert study below, experts also judged that the tests check all the requirements in 81 of 85 answers. We have added the mutation study to the revised paper.

[X] D. Mallis, M. Wang, A. S. Karadeniz, E. Ricci, A. Kacem, and D. Aouada. Text-to-CAD Evaluation with CADTests. arXiv preprint arXiv:2605.07807, 2026.

Several aspects related to the dataset preparation seem to be unclear and lack real-world grounding, particularly the selection of CAD models, the definition of functional requirements, and the categorization of different mechanisms.

FunCADBench provides no CAD models. Each task states the functional requirements of a mechanism, and the agent builds the CAD model. Public CAD models (e.g., on Printables and GitHub) exist for many families but are made for 3D printing to other specifications. For example, public watch models have no alarm.

The 46 families come from off-the-shelf products, mainly in the McMaster-Carr catalog, and from everyday mechanisms. Their requirements follow what each product is used for, e.g., a winch must lift, hold and safely release a load (see also our reply on functional requirements below). The families span five axes of mechanical demand: motion primitives, topology, complexity, contact and geometric difficulty (Section 3.2, Table 1).

Every task has a design that passes its tests. Agents passed 40 tasks across our runs, including the Grok + history setting (Section 5.1), whose per-task results we have added to the revised paper. For five of the other six (Tasks 7, 9, 20, 45 and 46), a human completed the closest failed agent design. For Task 34, we adapted an openly licensed public ratchet design. All six pass the unchanged tests. We will release these designs with the benchmark, and we have added this description of the dataset to Section 3 and Appendix H.1 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer 4hcy (2/2)

The paper would benefit from an explanation grounded by CAD expertise ... For example, the authors could have conducted a human expert study showcasing the validity of the functional requirements and their form and assessing the alignment of the conducted evaluation with human experts.

As suggested, 17 experts with more than two years of CAD experience took part in a human study on 10 representative tasks. Each expert read the requirements and tests of five tasks and answered four yes/no questions. Each also judged six designs from other tasks, half passed and half failed by the harness, after watching the simulation replay.

QuestionYes
The requirements are reasonable85/85
They cover the main functions of the mechanism75/85
The test is reasonable83/85
The test checks all the requirements81/85
A design the harness passes meets the requirements46/51

On designs the harness fails, experts mostly judged the motion to look right, since violations such as a slight interpenetration do not show in an animation. The harness catches them from the recorded trajectories (see our reply to Reviewer 1FEV). We have added the expert study to the revised paper.

In which form the considered AI agents generate CAD models? Python CAD programs (e.g., CadQuery) or BReps or any other form?

The agents write Python programs with build123d, a CAD library on the same OpenCASCADE kernel as CadQuery, which build B-Rep solids. Each design is submitted as the Python source, a multi-body STEP assembly and its physical declarations, i.e., rigid bodies, joints, actuators and materials (Appendix A.1). Project Chrono simulates collision meshes derived from the B-Rep, and the geometry checks run on the B-Rep itself (Appendix D.3). We have named the library in Section 3.1 of the revised paper.

How is intended use, called as functional requirements, is formulated here? how this is close to functional requirements used in real design workflows?

Each task starts with a short natural-language request, followed by a list of functional requirements. A requirement states an observable behavior under a given input or load, often with a target value and, where needed, a dependence on earlier states. The task also states fixed constraints and permitted operations, and leaves the parts and architecture open (Appendix A.1). For example, the winch (Task 42) must lift a load through a reduction, hold it against a reverse load, and release it and lock again.

This matches how requirements are written in engineering design. Before concepts are generated, a design specification lists solution-neutral requirements with measurable targets, as in the requirements list of systematic design [1] and the functional requirements of axiomatic design [2]. We have added this explanation to Section 3 of the revised paper.


[1] G. Pahl, W. Beitz, J. Feldhusen, and K.-H. Grote. Engineering Design: A Systematic Approach. Springer, 2007.

[2] N. P. Suh. The Principles of Design. Oxford University Press, 1990.

Reviewer_4hcyOpenReview ↗

Response to the authors’ rebuttal

I thank the authors for their detailed response and the substantial additional work. The mutation study gives the quantitative check of test reliability that was missing, and detecting 39 of 40 faults with no false rejections is convincing. The expert study with 17 CAD practitioners is also a welcome addition, and it shows that the requirements and tests are reasonable and aligned with expert judgment. The clarifications on dataset construction, the requirement formulation grounded in systematic and axiomatic design, the CAD representation and the reference designs for all 46 tasks address my remaining concerns.

One clarification question: were the mutations introduced manually by the authors or generated by an agent? Please state this in the revised paper.

We sincerely thank the reviewer for the positive assessment and for engaging with our response.

Regarding the follow-up question, the mutations were generated by a coding agent under fault classes and rules that we fixed in advance, and we manually checked every mutation. The rules are a single change at one location, numeric shortfalls of 20–40%, and unchanged design notes. For each mutation, we inspected the file diff, the renders, a before/after 3D view and the corresponding simulation results to confirm that the intended change is the only one and that it breaks the stated function. All mutations were frozen before the harness was run. We have added these details to Section 5.1 and Appendix H.3 of the revised paper.

Reviewer_1FEVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies functional CAD design, where an LLM agent is prompted by a user to create CAD mechanisms that must satisfy functional requirements in physics simulation. The authors argue that LLMs may produce visually plausible assemblies that do not fulfill their intended functions, and that verifying function is difficult because the same requirement can be realized by different mechanisms. They propose a harness (FunCAD-Harness) that guides the agent by deriving candidate-specific simulation tests from the functional requirements and returning diagnostic feedback to drive design revision. They also propose a benchmark (FunCADBench) covering 46 mechanism families with diverse physical and functional demands. Experiments with frontier models show that the harness improves task success over agents given only the requirements or simulation tools, that diagnostic feedback is key to this gain compared with pass/fail outcomes, and that complex mechanisms remain challenging for current agents.

Critical strengths
  • The paper tackles a practical, real-world design problem :whether LLM-generated CAD mechanisms actually work, not just look plausible; and shows a clear improvement over simpler agent workflows.
  • The candidate-specific test binding is interesting: separating abstract test templates from their binding to each design lets the harness evaluate different mechanisms for the same task without imposing a particular architecture.
  • It introduces a diverse, open-ended benchmark covering 46 mechanism families, which is a valuable resource for evaluating LLM agents on functional design.
Critical weaknesses
  • The ablations show the full harness is effective but do not isolate candidate-specific test binding, one of the main contributions.
  • The method is practical but of limited novelty: it mainly combines known agentic techniques (generate–test–revise loops, agentic testing), even though it addresses an important problem.
  • There are no external baselines, such as a visual-feedback CAD revision loop like CADCodeVerify [1] or an adapted BesiegeField-style setup [2].
  • The harness comparison (Table 2) uses only 2 of the 4 models.
  • The final evaluation relies on an LLM agent to configure the tests, with no validation against human experts.
  • Baseline designs require extra adaptation for evaluation, so some failures may come from incomplete testing rather than actual mechanical failures. This could affect the comparison with the full harness.
  • Generalization beyond the benchmark is unclear. Only six physical prototypes are shown, and the ability to handle new user requests is mainly demonstrated through the cup example.
  • The harness implementation and executable benchmark tests are not released, and no release plan is stated, which limits reproducibility.

References:

[1] Alrashedy et al. (2025). Generating CAD Code with Vision-Language Models for 3D Designs. ICLR 2025. https://arxiv.org/abs/2410.05340

[2] Zhang et al. (2026). Compositional Machine Design as Program Synthesis with LLMs. EMNLP 2026. https://arxiv.org/abs/2510.14980

Questions for authors
  • Will the harness, benchmark tasks, tests, and evaluation scripts be released?
  • Can you add an ablation comparing independent test binding with test binding performed by the constructor itself?
  • Why was no external baseline included?
  • Why does the harness comparison use only 2 of the 4 models?
  • How reliable is the LLM-configured final evaluation? Was it checked against human experts on a subset?
  • How many baseline failures come from incomplete adaptation or evaluation rather than mechanical failure?
  • How many successful tasks involved agent-adjusted operating parameters (load, duration...), and do they still pass under the original targets?
  • Were the printed prototypes fabricated directly from the agent’s CAD output, or manually modified?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 1FEV (1/4)

We thank the reviewer for their time and effort in reviewing our paper and for the constructive feedback.

The ablations show the full harness is effective but do not isolate candidate-specific test binding, one of the main contributions.

We thank the reviewer for pointing out this important issue. We agree that the ablations in the paper do not isolate test binding. Without binding, a template cannot run, and agents are left with their own checks, i.e., the simulation-tools setting in Table 2. We therefore isolate who binds: we let each constructor bind its own designs instead of the independent binder.

From the main experiment we took, for each of the four constructors, the last design that the independent binder had bound with a test. 155 of the 184 runs have one. The other 29 ended without any design receiving a test binding. We left out 16 designs that fail the harness's current geometry-connectivity check and had not passed in the paper. This leaves 139 designs over all 46 tasks (41 built by Fable, 25 by Kimi, 37 by Astra, 36 by Grok), with both passing and failing designs included.

Each design was bound twice: by the independent binder, and by its own constructor, which was given the record of its construction session but not the independent binding. Templates, simulator and assertions are unchanged, so only the binding differs. Four self-bindings timed out, leaving 135 designs. The two bindings agree on 126 of them (table below).

ConstructorDesignsAgreementSelf passes, independent failsSelf fails, independent passes
Fable3938/39 (97%)01
Kimi2320/23 (87%)21
Astra3734/37 (92%)12
Grok3634/36 (94%)02
All135126/135 (93%)36

We inspected the nine disagreements:

  • Self passes, independent fails (3). The constructor had failed to get all three accepted during its own run.
    • Cube and lid: the self-binding skipped requirements the simulation does not test (the cube's layers turn about one axis only, and the lid's retention was tested in one opening on an idealized hinge).
    • Pedal bin: the constructor loosened the test with a wider tolerance taken from its own development runs.
  • Self fails, independent passes (6). The two bindings measured the design differently. In one case, the self-binding sampled load power after each integration step, so the power turned slightly positive at stroke reversal and failed the "load only dissipates" check.

Self-binding thus errs in both directions. It lets failed designs through, likely because its construction context biases its decisions, and rejects others through its own measurement choices. In the loop, a wrong pass would stop revision on a failed design, and a wrong failure would send the constructor after a fault that does not exist. Here each constructor binds only once, after its run. Inside the loop, it would bind at every revision while trying to pass, with more chances to skip or loosen a requirement.

The wrong passes match what is known about models checking their own work: they favor their own outputs [3, 5] and, given access to the tests, change them to pass [6]. Agent systems therefore separate the agent that does the work from the agent that checks it, e.g., an independent test designer in code generation [4] and a separate evaluator in long-running app development [3]. Independent test binding brings this separation to physical tests, which must be bound to each new design. We have added this ablation to Section 5.2 and Appendix H.5 of the revised paper.

Generalization beyond the benchmark is unclear. Only six physical prototypes are shown, and the ability to handle new user requests is mainly demonstrated through the cup example.

FunCAD-Harness involves no training, so every benchmark task is a new request to it. Its ability to handle new requests is therefore measured on all 46 tasks, and the cup example is one more such request. The benchmark cannot include every possible request, but the cup task draws on the same five axes of mechanical demand (Table 1) that the benchmark covers. A request outside the benchmark adds one step, writing its test template: the constructor drafts it, and the independent evaluation agent checks it before binding (Appendix A.5).

The six printed prototypes are qualitative examples that show agent designs can be printed and operated (Section 5.5, Figure 9 and Appendix G). We have revised Section 1 to make this clear.

Were the printed prototypes fabricated directly from the agent's CAD output, or manually modified?

The six prototypes in Appendix G were printed directly from the STEP/mesh files exported by the agent's CAD scripts, and nobody redesigned their functional geometry. Our part was limited to normal print preparation (slicing, adding and removing supports) and assembling the printed parts.

AuthorsOpenReview ↗

Response to Reviewer 1FEV (2/4)

The method is practical but of limited novelty: it mainly combines known agentic techniques (generate-test-revise loops, agentic testing), even though it addresses an important problem.

We agree that generate-test-revise loops and agentic testing are known, and we do not claim them. Our contributions are:

  • A new problem. To our knowledge, this is the first work that turns a functional demand into a CAD assembly verified by physical tests in simulation. FunCADBench covers 46 mechanism families along five axes of mechanical demand. The problem matters both for applications and for understanding what LLM agents can do in the physical world.
  • Test binding. A revise loop needs a test, but a physical test depends on the design itself, so it cannot be written in advance. We therefore split testing into a fixed test template per task and concrete physical tests bound to each design once it exists. The self-binding ablation in our reply on the ablations shows that binding by an independent agent matters.
  • Insights (Table 2):
    • Simulation is necessary. Designs that look right often fail physically, and agents given only the requirements rarely produce working mechanisms.
    • A simulator alone is not enough. Left to test on their own, agents check parts of a mechanism and miss the full operation. Tests organized around the requested functions, with diagnostic feedback, roughly double success.
    • The bottleneck is physical reasoning, not vision. Strong visual 3D modeling does not lead to better functional design, and removing the constructor's visual access barely changes success.

There are no external baselines, such as a visual-feedback CAD revision loop like CADCodeVerify [1] or an adapted BesiegeField-style setup [2].

We have added an external baseline adapted from BesiegeField [2]. CADCodeVerify [1] does not apply to our tasks, as explained below.

BesiegeField builds machines from blocks in a game simulator. We kept its full agent workflow and replaced only the game-specific parts. Blocks became build123d parts with joints, and the game score became our formal evaluation, which reports the passed tests and measured values as the score does. Both methods use GPT-6 Astra and a 3-hour limit per task. To fit this limit, we ran 2 candidates with one refinement round instead of the default 8 candidates.

MethodTasks solved (/46)
BesiegeField [2], adapted9
FunCAD-Harness22

FunCAD-Harness solves 22 tasks, including all 9 solved by the baseline. Of the baseline's 37 failures, 30 are designs that fail the functional tests. We see two likely reasons:

  • Simulation access. In BesiegeField, a design is simulated only when it is scored. Our constructor simulates and probes its design as it builds, so it can find and fix functional failures before submission.
  • A lighter harness. BesiegeField fixes the workflow in advance: a meta-designer splits the machine, designers build it structure by structure, and separate roles inspect and refine it. Such scaffolding was designed for weaker models and can constrain frontier ones. FunCAD-Harness fixes only the tests and leaves the workflow to a single constructor with CAD and simulation tools. This matches recent reports that frontier models need less scaffolding [3].

CADCodeVerify refines a single part until its render matches a text description. It has no joints, actuation or assemblies, and its feedback is visual. Our results suggest that visual feedback adds little here. Many assemblies that look right still fail (Section 5), and removing the constructor's visual access barely changes success (Table 2).

We have added this baseline to Table 2 and the adaptation details to Appendix H.6.

The harness comparison (Table 2) uses only 2 of the 4 models.

We agree that running all four models would make Table 2 stronger. Each setting in Table 2 is a separate run over all 46 tasks at up to three hours per task, so adding Fable and Kimi means eight more full runs. This is costly, and we may not be able to complete all of them within the discussion period.

Astra and Grok cover the two verification styles in Section 5.4: Astra relies more on simulation and geometric checks, and Grok, like Fable, on reading requirements and code. The two models show consistent results. Both improve from requirements only to simulation tools to the full harness, and both drop with binary feedback. We have started running Fable and Kimi on a subset of the tasks and will add the results to the final version.

AuthorsOpenReview ↗

Response to Reviewer 1FEV (3/4)

How reliable is the LLM-configured final evaluation? Was it checked against human experts on a subset?

Good question! The LLM has a narrow role in the final evaluation. For the 46 tasks, the test templates are fixed in advance and checked by the authors, and the pass/fail assertions run deterministically on the recorded simulation trajectories. The LLM only binds the template interfaces to each candidate's components. We checked this step in two ways.

Agreement across binders. We kept the designs fixed and re-ran only the binding. In fresh sessions, Fable 5.1 and Kimi k3 bound the tests from scratch for the 46 final designs of the Fable run, without seeing the original Astra binding (41 designs are comparable). A fresh Astra session also re-bound 18 designs, each under two parameter sets.

ComparisonAgreement
Kimi vs. original Astra binding39/41 (95%)
Fable vs. original Astra binding38/41 (93%)
Kimi vs. Fable40/41 (98%)
Repeated Astra vs. original Astra binding33/36 (92%)

Across models, the binders disagree on only three designs.

Human experts. 17 experts, all with more than two years of CAD experience, each judged six designs from 10 tasks, three passed and three failed by the harness. After watching the simulation replay, they judged whether each design meets its requirements.

  • Passed designs. Experts agree with the harness in 46 of 51 judgments.
  • Failed designs. Experts rated them as working in 47 of 51 judgments. They look right in the replay, and most miss a requirement by too little to see, e.g., a slight interpenetration, or a rope paying out at 53.7 mm/s against a 45 mm/s limit.

Catching designs that look right but do not work is the goal of the paper, so the harness computes its results from the recorded trajectories. Our reply to Reviewer 4hcy also reports a mutation study of the tests themselves. We have added these studies to the revised paper.

How many successful tasks involved agent-adjusted operating parameters (load, duration...), and do they still pass under the original targets?

Of the 87 successful designs, 23 passed with operating parameters adjusted by the constructor. Two of them relaxed an acceptance threshold, and the other 21 kept every functional target. Counting only designs that pass with their original parameters leaves the paper's conclusions unchanged.

Each task fixes its functional targets and lets the constructor adjust operating parameters, such as actuator gain or test duration, within a permitted scope. The independent binder applies each change and records the original value, the new value and the reason (Appendix A.4). Where a prompt states numeric targets, all passes were tested at the stated values. For the other tasks, the closest record of the original parameters is each design's first formal test. We re-ran the 23 designs with those parameters, keeping each design and its binding unchanged:

Type of adjustmentDesignsStill pass with original parameters
Actuator gain or damping only80
Duration, force limit or travel83
Relaxed acceptance threshold20
Rejected by the current geometry check5-
Total233

Most of the failed re-tests come from the test conditions, not the targets. Operating parameters are test conditions, and like the rest of the test they depend on the design. In Task 3, for example, the first speed-control gain made the input reverse repeatedly. The constructor lowered the gain and lengthened the observation window, while the input speed, torque limit, sweep and return-error targets stayed the same (Appendix A.4). The two threshold designs miss the original limit narrowly, and the conservative count below does not accept them.

Accepting only designs that pass with their original parameters gives:

ModelIn the paperOriginal parameters only
Astra22/4618/46
Kimi14/469-11/46
Fable32/4626-27/46
Grok19/4614-17/46

The ranges depend on how we count these five designs, and on one design whose re-test exceeds the simulation time limit. The ranking of the models is unchanged, and the full harness still solves more tasks than the requirements-only, simulation-tools and binary-feedback settings and the adapted BesiegeField in Table 2. The revised paper reports both counts (Section 5.1, Appendix H.7).

Reviewer_7iCGOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors study whether coding agents can design mechanisms (represented as CAD programs) that achieve intricate physical goals in simulation, such as a ratchet or bar linkage. This is composed of FunCAD-Bench, which tests agents on 46 mechanism types, and FunCAD-Harness which iteratively LLM-edits and simulator-verifies a program. Commercial coding agents achieve between 0% and 50% on this task, and the harness enables them to avoid invalid mechanisms.

Critical strengths

The benchmark has impressively diverse and challenging tasks and reveals interesting failures of current coding agents on a very important capability.

The paper is very well written, and provides an appendix with a manifest of what is in its benchmark and prompts to reproduce its harness.

Critical weaknesses

The benchmark could reasonably be criticized for being simulated rather than real, however, until the agents pass a simulated benchmark this seems acceptable to me, as the real world will most likely be harder. Fig1 has an example of a 3D printed mechanism but Sec6's fabricatino paragraph doesnt mention making it or its success.

It would be good to repeat experiment 5.1 for more than one rollout but I do not expect this due to cost.

Questions for authors

Was the design in Figure 1(d) fabricated and functional, or is it illustrative?

While LLMs currently dont come close to saturating the benchmark, is there a risk or possibility that some tasks wont be reasonably solvable or will reject valid solutions, or that a fraction of the tasks would be harder than the real world? The presence of a few unsolvable tasks has been common in other benchmarks.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 7iCG

We thank the reviewer for acknowledging our contributions and for the constructive comments.

Was the design in Figure 1(d) fabricated and functional, or is it illustrative?

It is fabricated and functional. The cup-access mechanism in Figure 1(d) is a working 3D-printed prototype. Turning its single handle drives two side linkages that lift the rear cup above the front row while keeping it upright (Figure 9c and Appendix G). The video in our supplementary material shows all six printed mechanisms in operation, including this one.

We agree that the paper should have stated this clearly. In the revised paper, the Figure 1 caption now states it and points to Section 5.5, Appendix G and the video.

It would be good to repeat experiment 5.1 for more than one rollout but I do not expect this due to cost.

We agree that more rollouts would strengthen Section 5.1, and cost is what has held us back. In each run, the constructor gets up to three hours of active working time per task, spent on LLM turns, B-Rep compilation, and repeated Chrono simulations.

We have repeated the full harness for one model. Grok ran eight more rollouts over all 46 tasks, each without access to the others (Appendix F.6):

Grok, full harnessTasks solved (/46)
Run in the paper (Table 8)19
Each of the 8 repeated runs15-18
Any of the 8 repeated runs24
Grok + history (8 attempts with shared records)36

Single runs solve 15 to 19 tasks, a spread smaller than the gaps between the full harness and the baselines in Table 2, e.g., 10 tasks with simulation tools against 19 with the full harness. All four constructors also solve the tasks in a shared order of difficulty (Figure 6a).

Additional attempts help most when they share records. Eight independent runs together solve 24 tasks, mostly tasks that a single run already solves some of the time. Eight attempts that keep the feedback from earlier attempts solve 36.

While LLMs currently don't come close to saturating the benchmark, is there a risk or possibility that some tasks wont be reasonably solvable or will reject valid solutions, or that a fraction of the tasks would be harder than the real world?

Every task is known to be solvable under the tests as written, and we measured false rejections directly.

Solvability. None of the 46 task families is invented. They come from off-the-shelf products, mainly in the McMaster-Carr catalog, and from everyday mechanisms. Their requirements follow what each product is used for, e.g., a winch must lift, hold and safely release a load. So every task has at least one known physical realization, and the tests accept such designs. Agents passed 40 tasks across our runs, including the Grok + history setting (Section 5.1), whose per-task results we have added to the revised paper. For five of the other six (Tasks 7, 9, 20, 45 and 46), a human completed the closest failed agent design. For Task 34, we adapted an openly licensed public ratchet design. All six pass the unchanged tests (Appendix H.1).

The manual builds also showed that the gap between the best agent attempt and a passing design is often small. In Task 20, for example, the closest candidate failed only because two balls slightly overlapped where they stacked off-axis in a loose magazine. Narrowing the magazine so that the balls stack in line made it pass. Across these builds we found no requirement that contradicts itself or is physically impossible to meet.

False rejections. The tests are solution-neutral templates bound to each candidate. In the mutation and re-binding studies (see our replies to Reviewers 4hcy and 1FEV), none of the 10 unchanged control designs was rejected, and three different LLMs binding the same 41 designs disagree on only three.

Some tasks may be harder in simulation than in the real world, e.g., where contact resolution is sensitive. We will release the per-task near-miss analysis and flag such tasks in the benchmark.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

General response to all reviewers

We thank all reviewers for their time and efforts put into the reviewing process and their constructive comments. We have uploaded a revised paper. The main changes are summarized below, and each reply gives the details. All section, figure and table numbers refer to the revised paper.

New experiments

  • Reliability of the tests and the final evaluation (1FEV, 4hcy, 7iCG).
    • Mutation study: the tests caught 39 of 40 faults injected into passing designs and rejected none of 10 unchanged designs.
    • Re-binding: three different LLMs binding the same 41 designs disagree on only three.
    • Expert study: 17 experts with more than two years of CAD experience judged that the tests check all the requirements in 81 of 85 answers, and agreed with the harness in 46 of 51 judgments of passed designs.
  • Value of independent test binding (1FEV, Section 5.2). When each constructor binds its own designs, it passes some that skip or loosen requirements.
  • Baselines (1FEV).
    • External baseline (Table 2): an adapted BesiegeField solves 9 of 46 tasks, against 22 for FunCAD-Harness with the same model and time limit.
    • Failure analysis: every failed baseline run fails because of a design defect, not incomplete testing.
  • Repeated runs (7iCG, Appendix F.6). Eight more independent runs of Grok with the full harness solve 15 to 18 tasks each, close to the 19 in the paper.
  • Solvability (7iCG, 4hcy). For the six tasks no agent passed, we completed reference designs that pass the unchanged tests, so every task is solvable. We also added per-task results of Grok + history (Table 8 in Appendix E.4).

Changes to the text

  • Dataset and requirements (4hcy, Section 3). We added where the task families come from, how the functional requirements are written, and how they relate to requirement lists in engineering design.
  • CAD representation (4hcy, Section 3.1). We now name build123d, the Python CAD library the agents use to build B-Rep assemblies.
  • Operating parameters (1FEV, Section 5.1, Appendices A.4 and H.7). We state which parameters the constructor may adjust, and add a conservative count that accepts only designs passing with their original parameters.
  • Fabricated prototype (7iCG, Figure 1). The Figure 1 caption now states that Figure 1(d) is a working 3D-printed prototype, and points to Section 5.5, Appendix G and the supplementary video.
  • Printed prototypes (1FEV, Section 1). Section 1 now presents the cup example as a qualitative case study.
  • Language. We polished the language and flow of the whole paper.

An anonymized repository with the harness, the task specifications and the executable test templates is available at https://anonymous.4open.science/r/funcad-harness-review-BBB4/.

We hope these additions address the reviewers' concerns, and we are happy to answer further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.