- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CADWorld, a benchmark for evaluating long-horizon computer-use agents in mechanical CAD workflows. It contains 200 FreeCAD tasks across 11 categories, including sketching, part modeling, assembly, CAM, and FEM. Agents interact with the FreeCAD GUI through screenshots and actions, and their outputs are evaluated using task-specific executable checks on saved native CAD artifacts rather than visual similarity alone. The authors evaluate seven computer-use agents and report that the strongest achieves only 17.5% success, highlighting the challenges of reliable CAD automation and the gap between general GUI capabilities and engineering workflow requirements.
Critical strengths
- Broad coverage of CAD workflows. CADWorld covers 200 tasks across 11 mechanical CAD categories, including modeling, assembly, CAM, and FEM. This provides a valuable benchmark for evaluating agents on diverse engineering tasks beyond basic CAD generation.
- Artifact-grounded evaluation. The benchmark evaluates saved native CAD files using task-specific executable checks, rather than relying only on visual similarity. This is particularly valuable for CAD applications, where geometric accuracy, parametric structure, and engineering constraints are essential.
Critical weaknesses
- Insufficient comparison with prior CAD GUI agent research. The paper does not discuss closely related work such as VideoCAD, which has already studied long-horizon CAD UI interactions. Moreover, the experiments mainly evaluate general-purpose computer-use agents without including CAD-specific GUI agent baselines. This weakens the paper's novelty positioning and the completeness of its evaluation.
- Limitations in evaluation design. The benchmark relies on task-specific evaluators, but their scalability and ability to accommodate different valid CAD modeling approaches are not sufficiently discussed. Moreover, the strict binary scoring does not capture partial task completion, making it difficult to interpret the reported low success rates and distinguish near-complete results from complete failures.
Questions for authors
- Comparison with prior work: VideoCAD has already explored long-horizon CAD GUI interactions and CAD-specific agent learning. Could the authors clarify how CADWorld differs from VideoCAD and why CAD-specific GUI agents were not included in the evaluation?
- Evaluation metrics: Since the benchmark uses strict binary success criteria, could the authors report check-level pass rates or partial completion scores? This would help clarify whether the low success rates reflect fundamental task failures or near-complete results that fail only a few requirements.