- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces EngiWorld, a benchmark designed to evaluate the capabilities of autonomous agents in professional engineering environments. The benchmark consists of 1,301 expert-curated tasks across six engineering domains (CAD, CAE, CAM, BIM, EDA, and 3D visualization) and 26 professional software platforms. It covers six task categories, including single-software execution, multi-software workflows, software selection, quantitative design, image-based modeling, and open-ended tasks, supporting both GUI and CLI interactions.
A key component of EngiWorld is its artifact-centric evaluation framework, which employs domain-specific programmatic verifiers to assess engineering deliverables based on geometric validity, physical feasibility, and task-specific constraints. For quantitative design tasks, the framework additionally provides continuous quality scores subject to feasibility requirements.
The authors evaluate seven frontier models on a stratified subset of 300 tasks. Results reveal substantial limitations in current agents' engineering capabilities, with the best-performing model achieving an EngiScore of 44.3 and only 3.6% of multi-software attempts succeeding. Additional ablation studies and failure analyses investigate the effects of observation configurations, interaction history, and common execution failures.
Critical strengths
-
Practical Relevance: The paper addresses an important problem by evaluating AI agents in realistic professional engineering environments rather than relying on simplified or purely text-based tasks.
-
Broad Coverage: EngiWorld includes 1,301 tasks across six engineering domains and 26 software platforms, providing a comprehensive benchmark for assessing engineering agents.
-
Artifact-Centric Evaluation: The use of domain-specific verifiers to evaluate actual engineering deliverables is valuable and more meaningful than relying solely on LLM-based judgments.
-
Comprehensive Experiments: The evaluation of seven frontier models, together with ablation studies and failure analyses, provides useful insights into the current limitations of engineering agents.
Critical weaknesses
From my sides, this paper has two main problems.
-
Limited Differentiation from Existing Evaluation Methodologies: although the paper provides a comparsion with existing benchmarks, including [1]Agents' Last Exam (ALE)(NIPS 2026), the comparison primarily focuses on task coverage and supported workflow capabilities. It does not sufficiently clarify the methodological novelty of the proposed artifact-centric evaluation framework. Also in [2]EngDesign (NeurIPS 2025), the author has used the feasibility-gated continuous scoring, including in tasks involving commercial software. The authors should more explicitly distinguish their verification and scoring methodology from these existing approaches. Without such clarification, it remains unclear whether the primary contribution lies in a fundamentally new evaluation methodology or in the scale and engineering-specific specialization of existing benchmark frameworks.
-
The validation coverage appears to be quite limited, with only 7.4% of the tasks included in the systematic verifier audit. Compared with benchmarks such as Agents' Last Exam (ALE), I do not see sufficient evidence demonstrating the reliability of the remaining verifiers or explaining how their validity is ensured.
Furthermore, the negative tests primarily consider two relatively simple cases: submissions without engineering artifacts and submissions containing only numerical metrics. I do not think these tests are sufficient to demonstrate whether the verifiers can identify subtle but genuine engineering errors. For example, a CAD model may have the correct overall appearance but contain an incorrect internal hole diameter. Can the verifier reliably detect such geometric inconsistencies? I believe evaluating these challenging cases is substantially more important than simply checking for missing artifacts. Conversely, an agent might generate an engineering-correct CAD model using a different internal data structure or representation from the reference solution, yet receive a score of zero. The authors have not provided sufficient evidence to rule out such false rejections.
- I think selecting choice as a difference contribution is overclaimed. The paper emphasizes software selection as a distinctive capability of engineering agents, but its practical relevance is not sufficiently justified. In many real-world engineering environments, software tools are largely predetermined by established industrial workflows, licensing constraints, and domain-specific requirements. Therefore, it is unclear how frequently agents would need to independently select among alternative engineering software platforms in realistic professional settings. More importantly, although the paper introduces software selection as a distinct capability, the evaluation does not clearly disentangle tool-selection errors from downstream execution failures. An agent may select an appropriate software platform but still fail due to difficulties in operating it or producing the required artifacts. Consequently, the reported task success rates may not accurately reflect the agents' software-selection capabilities. This raises concerns about both the practical relevance and construct validity of this task category. However, considering that Choice accounts for only about 10.8% of all the tasks in EngiWorld, it is not a major weakness.
[1] Sun et al., “Agents’ Last Exam,” NeurIPS, 2026. https://arxiv.org/abs/2606.05405
[2] Guo et al., “Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs,” NeurIPS, 2025. https://openreview.net/forum?id=Wmsnx7EPel
Questions for authors
- The systematic verifier audit covers only 96 out of 1,301 tasks. How do the authors ensure the reliability of the remaining verifiers? Have they evaluated false positives and false negatives involving subtle engineering errors or semantically equivalent solutions with different representations? Or do we have more human evaluations/feedbacks?
- For software-selection and cross-software tasks, how do the authors distinguish failures caused by incorrect engineering decisions from those caused by software execution difficulties or strict workflow and submission requirements? Can the authors provide evidence that these tasks reflect realistic engineering workflows?
- Given that EngDesign [2] and Agents' Last Exam [1] already explore engineering verification, professional software workflows, and feasibility-gated scoring, what are the key methodological advances of EngiWorld beyond broader task and software coverage?
A additional concern: This concern is also related to my first point. The overall task design appears somewhat similar to Agents' Last Exam (ALE). Given that ALE incorporates detailed human expert feedback and evaluation procedures, how do the authors ensure that EngiWorld's automated evaluation is equally reliable or more trustworthy? Is there sufficient human expert validation to support the correctness of its evaluation results?
2 comments
Thank you for your thoughtful feedback and for raising your score. We sincerely appreciate your suggestions and will take them into consideration in our future work.
The authors have addressed my concerns, and I have decided to raise my score to 3. However, after reviewing some of the example workflows in Appendix D.1, I still question the practical significance of the problem addressed in this paper for real-world engineering applications, such as CAD. In practice, most companies maintain their own libraries of reusable components, and engineering workflows often involve assembling existing components and performing validation or verification, rather than designing and analyzing individual structures from scratch.
Therefore, I would encourage the authors to consider more complex, assembled structures involving multiple interacting components, rather than focusing solely on the analysis of isolated structures. I believe this direction could make the proposed approach more relevant and valuable for practical engineering applications in the future. Recent advances in agent, such as codex/cc have demonstrated promising capabilities in generating complex CAD models. However, automatically verifying assembled structures while ensuring their overall structural integrity, functional consistency, and engineering feasibility remains a significant challenge. I know it is more time consuming, but it is more pratical.