acceptodds
Under review as a conference paper at ICLR 2027

EngiWorld: What Can Frontier Agents Deliver in Professional Engineering Environments?

Abstract

Autonomous agents have made rapid progress in general-purpose computer use, but reliable automation of professional industrial engineering remains out of reach, as engineering workflows demand reasoning over geometric and physical constraints and dependencies preserved across software and design stages. We present **EngiWorld**, the first benchmark structured around the complete design loop: 1,301 expert-curated tasks spanning 6 engineering domains (CAD, CAE, CAM, BIM, EDA, and 3D visualization) and 26 professional software platforms, with both GUI and CLI interfaces and 6 task types ranging from software-selection to open-ended tasks. We further introduce an artifact-centric evaluation methodology built on a unified domain-verifier suite, which programmatically checks the geometric validity, physical feasibility, and rule compliance of final and intermediate artifacts, and scores quantitative design tasks continuously by specification attainment rather than binary success. Evaluation of seven frontier models reveals a substantial capability gap: the strongest model achieves an EngiScore of only 44.3, and just 3.6% of multi-software attempts succeed. EngiWorld provides the first rigorous foundation for measuring progress toward agents that operate professional engineering software end to end.

Primary area:
datasets and benchmarks
Keywords:
Agent Evaluation, Engineering Agent, Computer-Use Agent, Engineering Design

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1KnyOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EngiWorld, a benchmark designed to evaluate the capabilities of autonomous agents in professional engineering environments. The benchmark consists of 1,301 expert-curated tasks across six engineering domains (CAD, CAE, CAM, BIM, EDA, and 3D visualization) and 26 professional software platforms. It covers six task categories, including single-software execution, multi-software workflows, software selection, quantitative design, image-based modeling, and open-ended tasks, supporting both GUI and CLI interactions.

A key component of EngiWorld is its artifact-centric evaluation framework, which employs domain-specific programmatic verifiers to assess engineering deliverables based on geometric validity, physical feasibility, and task-specific constraints. For quantitative design tasks, the framework additionally provides continuous quality scores subject to feasibility requirements.

The authors evaluate seven frontier models on a stratified subset of 300 tasks. Results reveal substantial limitations in current agents' engineering capabilities, with the best-performing model achieving an EngiScore of 44.3 and only 3.6% of multi-software attempts succeeding. Additional ablation studies and failure analyses investigate the effects of observation configurations, interaction history, and common execution failures.

Critical strengths
  1. Practical Relevance: The paper addresses an important problem by evaluating AI agents in realistic professional engineering environments rather than relying on simplified or purely text-based tasks.

  2. Broad Coverage: EngiWorld includes 1,301 tasks across six engineering domains and 26 software platforms, providing a comprehensive benchmark for assessing engineering agents.

  3. Artifact-Centric Evaluation: The use of domain-specific verifiers to evaluate actual engineering deliverables is valuable and more meaningful than relying solely on LLM-based judgments.

  4. Comprehensive Experiments: The evaluation of seven frontier models, together with ablation studies and failure analyses, provides useful insights into the current limitations of engineering agents.

Critical weaknesses

From my sides, this paper has two main problems.

  1. Limited Differentiation from Existing Evaluation Methodologies: although the paper provides a comparsion with existing benchmarks, including [1]Agents' Last Exam (ALE)(NIPS 2026), the comparison primarily focuses on task coverage and supported workflow capabilities. It does not sufficiently clarify the methodological novelty of the proposed artifact-centric evaluation framework. Also in [2]EngDesign (NeurIPS 2025), the author has used the feasibility-gated continuous scoring, including in tasks involving commercial software. The authors should more explicitly distinguish their verification and scoring methodology from these existing approaches. Without such clarification, it remains unclear whether the primary contribution lies in a fundamentally new evaluation methodology or in the scale and engineering-specific specialization of existing benchmark frameworks.

  2. The validation coverage appears to be quite limited, with only 7.4% of the tasks included in the systematic verifier audit. Compared with benchmarks such as Agents' Last Exam (ALE), I do not see sufficient evidence demonstrating the reliability of the remaining verifiers or explaining how their validity is ensured.

Furthermore, the negative tests primarily consider two relatively simple cases: submissions without engineering artifacts and submissions containing only numerical metrics. I do not think these tests are sufficient to demonstrate whether the verifiers can identify subtle but genuine engineering errors. For example, a CAD model may have the correct overall appearance but contain an incorrect internal hole diameter. Can the verifier reliably detect such geometric inconsistencies? I believe evaluating these challenging cases is substantially more important than simply checking for missing artifacts. Conversely, an agent might generate an engineering-correct CAD model using a different internal data structure or representation from the reference solution, yet receive a score of zero. The authors have not provided sufficient evidence to rule out such false rejections.

  1. I think selecting choice as a difference contribution is overclaimed. The paper emphasizes software selection as a distinctive capability of engineering agents, but its practical relevance is not sufficiently justified. In many real-world engineering environments, software tools are largely predetermined by established industrial workflows, licensing constraints, and domain-specific requirements. Therefore, it is unclear how frequently agents would need to independently select among alternative engineering software platforms in realistic professional settings. More importantly, although the paper introduces software selection as a distinct capability, the evaluation does not clearly disentangle tool-selection errors from downstream execution failures. An agent may select an appropriate software platform but still fail due to difficulties in operating it or producing the required artifacts. Consequently, the reported task success rates may not accurately reflect the agents' software-selection capabilities. This raises concerns about both the practical relevance and construct validity of this task category. However, considering that Choice accounts for only about 10.8% of all the tasks in EngiWorld, it is not a major weakness.

[1] Sun et al., “Agents’ Last Exam,” NeurIPS, 2026. https://arxiv.org/abs/2606.05405
[2] Guo et al., “Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs,” NeurIPS, 2025. https://openreview.net/forum?id=Wmsnx7EPel

Questions for authors
  1. The systematic verifier audit covers only 96 out of 1,301 tasks. How do the authors ensure the reliability of the remaining verifiers? Have they evaluated false positives and false negatives involving subtle engineering errors or semantically equivalent solutions with different representations? Or do we have more human evaluations/feedbacks?
  2. For software-selection and cross-software tasks, how do the authors distinguish failures caused by incorrect engineering decisions from those caused by software execution difficulties or strict workflow and submission requirements? Can the authors provide evidence that these tasks reflect realistic engineering workflows?
  3. Given that EngDesign [2] and Agents' Last Exam [1] already explore engineering verification, professional software workflows, and feasibility-gated scoring, what are the key methodological advances of EngiWorld beyond broader task and software coverage?

A additional concern: This concern is also related to my first point. The overall task design appears somewhat similar to Agents' Last Exam (ALE). Given that ALE incorporates detailed human expert feedback and evaluation procedures, how do the authors ensure that EngiWorld's automated evaluation is equally reliable or more trustworthy? Is there sufficient human expert validation to support the correctness of its evaluation results?

2 comments

The authors have addressed my concerns, and I have decided to raise my score to 3. However, after reviewing some of the example workflows in Appendix D.1, I still question the practical significance of the problem addressed in this paper for real-world engineering applications, such as CAD. In practice, most companies maintain their own libraries of reusable components, and engineering workflows often involve assembling existing components and performing validation or verification, rather than designing and analyzing individual structures from scratch.

Therefore, I would encourage the authors to consider more complex, assembled structures involving multiple interacting components, rather than focusing solely on the analysis of isolated structures. I believe this direction could make the proposed approach more relevant and valuable for practical engineering applications in the future. Recent advances in agent, such as codex/cc have demonstrated promising capabilities in generating complex CAD models. However, automatically verifying assembled structures while ensuring their overall structural integrity, functional consistency, and engineering feasibility remains a significant challenge. I know it is more time consuming, but it is more pratical.

Thank you for your thoughtful feedback and for raising your score. We sincerely appreciate your suggestions and will take them into consideration in our future work.

Reviewer_gDJuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a benchmark for evaluating LLM agents on engineering workflows across several fields (CAD, CAE, CAM, BIM, EDA and 3D visualization) and many software platforms. The benchmark includes workflows that span several applications. These test whether agents can produce outputs that remain usable as they move between software tools and engineering domains. Evaluation is programmatic: verifiers check the artifacts produced at different stages of the design process. The authors evaluate seven frontier AI models and analyze their performance and failure cases in depth. Overall, the results show that current frontier models fall well short when a task spans multiple stages of an engineering workflow.

Critical strengths
  • The benchmark’s scope matters a great deal to the community working on AI for engineering design. Existing benchmarks mostly evaluate isolated tasks and do not check whether the outputs remain usable at later stages of the engineering workflow. To the best of my knowledge, this benchmark is the first to address this.

  • The paper includes an extensive experimental study on the proposed benchmark. Seven frontier AI models are evaluated across the different task categories. Several ablations examine how the agents’ inputs affect performance, such as access to images and accessibility-tree information. The paper also analyzes the failure modes in detail.

Critical weaknesses

The main weakness of this paper is that it does not clearly explain how the tasks and the programmatic verifiers were built. This raises concerns about the reliability of the benchmark. The paper and the appendix repeatedly say that experts define the tasks and that AI then helps produce the final task specifications and verifiers, but they do not say how the work was split. What exactly did the experts do, and what did the AI do? How many experts were involved, and what are their fields of expertise? This matters because the benchmark covers very different environments, from BIM and CAD to CAM and CAE. For programmatic verification to be trustworthy, the paper should state clearly which parts came from experts and which came from AI. A useful way to do this would be to walk through a few concrete examples of the process: what the experts provided as input, how the verifiers were generated from it, and how the experts validated them.

Questions for authors
  • What exactly do experts do, and what AI does for generating the verifiers and the tasks?

  • How many experts were involved?

  • The experts are from which field, as this benchmark spans multiple environments from BIM, CAD, CAM, to CAE, etc?

  • Could the authors report how often the verifiers agree with expert judgments on a random sample of agent submissions?

Reviewer_JtVEOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark, EngiWorld, that encompasses 6 engineering domains, 26 specialized software platforms, and 1,301 tasks. EngiWorld accounts for software selection, cross-platform collaboration, quantitative design, and open-ended engineering tasks.

Critical strengths
  • This is a comprehensive benchmark for engineering agents with task design that more closely resembles actual engineering workflows.

  • The authors designed an automated verifier and introduced continuous scoring for quantitative tasks. This better aligns with engineering problem requirements than a simple task success rate.

  • The paper evaluates models across various tasks, software interaction, costs, and failure patterns. It also offers valuable ablation insights on image inputs and visual feedback.

Critical weaknesses
  • The paper claims to evaluate the complete Engineering Design Loop. But most tasks focus on a single software tool, while multi-software and open-ended tasks account for only 4.6% and 0.8%, respectively.

  • The current experiments lack an analysis of the different stages of the design process. It is unclear whether failures come from reasoning, software operation, or cross-software collaboration.

Questions for authors
  1. Table 2 shows that the performance of different models varies significantly between GUI and CLI environments. Can the authors analyze whether this is due to visual understanding, software operation, or task complexity?

  2. The current metric focuses on the final success rate. How does the agent use feedback during execution? For instance, can the agent identify wrong simulation settings or design constraints? Do agents leverage feedback and make iterative corrections?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.