acceptodds
Under review as a conference paper at ICLR 2027

CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided Design

Abstract

Computer-use agents are increasingly evaluated in realistic desktop environments, but existing benchmarks provide limited coverage of professional engineering workflows whose outputs are persistent, structured artifacts. Mechanical computer-aided design (CAD) is a particularly demanding setting: an agent must manipulate geometry and constraints over long interaction horizons while producing a native project whose dimensions, construction structure, and downstream engineering state remain valid. We introduce CADWorld, a benchmark for long-horizon computer use in FreeCAD. CADWorld contains 200 tasks spanning 11 mechanical-CAD workflow categories, including sketching, part modeling, assembly, CAM, FEM, measurement, mesh processing, and technical drawing. Agents operate through screenshots and GUI actions, while success is determined by task-specific executable checks over saved FreeCAD artifacts and auxiliary outputs, covering geometric properties, parametric structure, constraints, manufacturing state, and simulation results. Across seven current agents on the full benchmark, the strongest agent achieves 17.5% success, compared with an 87.0% expert reference pass. We find that weaker agents often fail before producing a valid artifact, whereas stronger agents increasingly fail on structural, geometric, and construction-process requirements. CADWorld therefore exposes a gap between general GUI competence and reliable execution of persistent, verifiable engineering workflows. Project accessible at https://anonymous.4open.science/r/ANONYMIZED

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Computer Use Agent, AI4Engineer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2t7JOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CADWorld, a benchmark for evaluating long-horizon computer-use agents in mechanical CAD workflows. It contains 200 FreeCAD tasks across 11 categories, including sketching, part modeling, assembly, CAM, and FEM. Agents interact with the FreeCAD GUI through screenshots and actions, and their outputs are evaluated using task-specific executable checks on saved native CAD artifacts rather than visual similarity alone. The authors evaluate seven computer-use agents and report that the strongest achieves only 17.5% success, highlighting the challenges of reliable CAD automation and the gap between general GUI capabilities and engineering workflow requirements.

Critical strengths
  1. Broad coverage of CAD workflows. CADWorld covers 200 tasks across 11 mechanical CAD categories, including modeling, assembly, CAM, and FEM. This provides a valuable benchmark for evaluating agents on diverse engineering tasks beyond basic CAD generation.
  2. Artifact-grounded evaluation. The benchmark evaluates saved native CAD files using task-specific executable checks, rather than relying only on visual similarity. This is particularly valuable for CAD applications, where geometric accuracy, parametric structure, and engineering constraints are essential.
Critical weaknesses
  1. Insufficient comparison with prior CAD GUI agent research. The paper does not discuss closely related work such as VideoCAD, which has already studied long-horizon CAD UI interactions. Moreover, the experiments mainly evaluate general-purpose computer-use agents without including CAD-specific GUI agent baselines. This weakens the paper's novelty positioning and the completeness of its evaluation.
  2. Limitations in evaluation design. The benchmark relies on task-specific evaluators, but their scalability and ability to accommodate different valid CAD modeling approaches are not sufficiently discussed. Moreover, the strict binary scoring does not capture partial task completion, making it difficult to interpret the reported low success rates and distinguish near-complete results from complete failures.
Questions for authors
  1. Comparison with prior work: VideoCAD has already explored long-horizon CAD GUI interactions and CAD-specific agent learning. Could the authors clarify how CADWorld differs from VideoCAD and why CAD-specific GUI agents were not included in the evaluation?
  2. Evaluation metrics: Since the benchmark uses strict binary success criteria, could the authors report check-level pass rates or partial completion scores? This would help clarify whether the low success rates reflect fundamental task failures or near-complete results that fail only a few requirements.
Reviewer_MNnxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

CADWorld is a computer-use benchmark built in FreeCAD with 200 tasks across 11 mechanical CAD workflow categories such as sketching, part modeling and assembly. The agent sees screenshots and sends GUI actions, and a task counts as solved when every task-specific executable check on the saved FreeCAD file and its auxiliary outputs passes. The paper evaluates seven agents, and the best, GPT5.4, still falls well short of the human expert reference.

Critical strengths

The paper provides its own evaluation protocol for computer-use agents on CAD. Compared with general computer-use benchmarks such as OSWorld, it checks CAD properties of the saved artifact, geometric properties, parametric structure, constraints, manufacturing state and simulation results, and it covers CAM and FEM alongside sketching and part modeling.

Critical weaknesses
  1. The measurement cannot be checked. The link in the abstract is a placeholder and there is no supplementary material, so none of the tasks, evaluators or recorded trajectories can be inspected. The paper reports "repeated manual checks" but not what they found. The expert reference that anchors the headline gap is a single expert reference pass, and who did it, with what qualification and whether they wrote the tasks are not stated, the ethics statement says the demonstrations were "collected for task construction".

  2. The benchmark cannot distinguish itself from existing ones, and the first contribution does not hold. OSWorld 2.0 already has a CAD subset, and VideoCAD [1] already studies long-horizon CAD UI interactions. The first contribution claims to formulate mechanical CAD as a computer-use problem, yet the paper delivers only a generic POMDP in Appendix C.1, of the same form as OSWorld's, and a claimed contribution belongs in the main text. ProSoftArena [2], which also has CAD tasks, already reported the main finding, low agent success in professional software. The CAD-specific difficulties the introduction names, geometric intent, state-dependent operations and dependencies over long horizons, are not measured directly.

References

  • [1] Man et al. VideoCAD: A Dataset and Model for Learning Long-Horizon 3D CAD UI Interactions from Video. arXiv:2505.24838.
  • [2] Ai et al. ProSoftArena: Benchmarking Hierarchical Capabilities of Multimodal Agents in Professional Software Environments. arXiv:2601.02399.
Questions for authors

Clarification requests

  1. Please provide a working anonymous repository with the tasks, evaluators and recorded trajectories. Please report how the evaluators were validated, including how they behave on artifacts known to be correct or wrong, and what went wrong on the tasks the expert failed.
  2. Who performed the expert reference pass, with what qualification, and did they write the tasks? Why are only 112 tasks recorded and none in assembly, CAM or FEM, while Figure 4 reports expert rates there? State this and release the recordings.
  3. How much of the ranking is interface, and how much is noise? GPT5.4 and Opus4.8 run through their vendors' computer-use tools with reasoning on, while the open models go through the paper's own adapter with thinking off or unavailable, and only one run per agent is reported (I expect the evaluation to be multiple runs with mean and standard deviation reported), so the top two differ by three tasks. Can you also run GPT5.4 or Opus4.8 through your adapter with thinking off, and repeat the top models two or three times on a subset?

Suggestions further

  1. I came across ComCADBench [3] and Workflow-GYM [4] on arXiv as well. Can you discuss how CADWorld differs from them?
  2. Rewrite the contribution list around what is delivered, the task set and the evaluators.
  3. I found it really hard to gather the necessary information from the paper. Please provide what a benchmark paper needs to be used or reproduced: an introduction to FreeCAD and its workbenches, the versions of FreeCAD and of the FEM solver and mesher, whether the runner builds on OSWorld's DesktopEnv as Figure 2 suggests, the sources and licenses of the tasks collected from tutorials and documentation, a full task list, what each evaluator tests, task by task, and the tolerance values, and a table of contents for the appendix.
  4. Measure the CAD-specific failures the introduction describes instead of the coarse failure taxonomy, which records only the first applicable type per episode.

References (continued)

  • [3] Ai et al. ComAct: Reframing Professional Software Manipulation via COM-as-Action Paradigm. arXiv:2606.13239.
  • [4] Zhu et al. Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields. arXiv:2606.11042.
Reviewer_RfmfOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CADWorld. It is a benchmark of 200 FreeCAD tasks across 11 mechanical-engineering workflow categories. The agents interact using screenshots, GUI actions, with careful considerations of specifics of UI (like draging and rotating objects, as shown in supp). Several models are evaluated as agents, and compared against a human operator to compare a number of actions. The key contribution is that persistent engineering artifacts are evaluated againts only screenshots of generated geometry.

Critical strengths
  1. The proposed benchmark coveres 11 engineering tasks, which is quite a lot. Most current benchmarks covers 1-3 engineering tasks.
  2. The benchmark focus on evalution of results with a number of checks, covering a lot of various properties of the result, not only geometric consistency.
Critical weaknesses
  1. I don't see significant scientific novelty behind this benchmark. It looks like a technical step ahead compared to current benchmarks.
  2. No modern models starting from GPT-6 and Claude Opus 5 have been evaluated, but these are the models which really step up 3D understanding, including CAD. So we don't know how saturated or useful this benchmark.
  3. The 87% success ratio of human operators without carefull analysis of the success rate raises quetions regarding evaluator of the benchmark. Is it a problem of human operators or evaluators are too strict? In engineering there are sometimes several ways to achieve the same result.
Questions for authors
  1. What is specific scientific novelty behind this benchmark?
  2. Why haven't modern models starting from GPT-6 and Claude Opus 5 have been evaluated?.
  3. What is the reason behind only 87% success ratio of human operators, if the benchmark is prepared by humans?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.