acceptodds
Under review as a conference paper at ICLR 2027

AeroCopilotBench: Safety-Gated Evaluation of LLM Agents on Aircraft Emergency Procedures in an Executable Cockpit

Abstract

Aviation knowledge question answering cannot directly assess the operational effectiveness and safety compliance of large language models throughout aircraft emergency procedures. We introduce AeroCopilotBench and its executable cockpit environment, ACOE, which define state-transition rules, task goals, and trajectory-level safety constraints based on aircraft-specific Pilot's Operating Handbooks (POHs). The benchmark comprises 12 scenario templates and 73 tasks across two aircraft types, evaluating task completion, safety compliance, execution discipline, and repeatability. Across repeated evaluations of 12 models, the highest safety-gated success rate is 72.6%. Most failed episodes achieve all critical goals but do not satisfy all remaining terminal goals. Across repeated runs, models still omit steps that they execute in other runs of the same task. Analysis of failed trajectories further reveals that some reasons for actions that conflict with the aircraft's POH recur across models. These results expose shortcomings in complete procedure execution, consistency across runs, and aircraft-specific emergency response.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agent benchmark, interactive environment, tool use, procedural task execution, safety evaluation, failure analysis, aviation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_DHEKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces AeroCopilotBench, a benchmark for evaluating how large language model agents execute aircraft emergency procedures in an executable cockpit. It covers the Cessna 172S and Piper PA-44-180 and systematically evaluates 12 models from different providers and model families. After receiving a task instruction from the Pilot Flying (PF), the models rely on their own aviation knowledge and feedback from the environment to carry out the procedures. The evaluation covers task completion, safety compliance, execution discipline, and repeatability, with safety-gated success rate as the primary metric. The paper also provides a relatively detailed analysis of failed executions. In my view, the benchmark contributes to research on large language model assistance in flight operations.

Critical strengths

1.To my knowledge, this study introduces the first benchmark for evaluating how large language model agents execute aircraft-specific emergency procedures in an interactive cockpit. It contributes to research on the use of large language models to assist flight operations. 2.The evaluation covers a range of widely used large language models, allowing their performance to be compared across model families.

Critical weaknesses

1.The paper does not clearly explain the significance of the proposed benchmark for the field or discuss its potential applications.

Questions for authors

1.The authors should discuss the research significance and potential implications of using large language models to carry out aircraft emergency procedures.

2.Why was the temperature set to 0.2 rather than 0 for all evaluated models? This choice may substantially affect the reported repeatability metric (pass³), which measures the proportion of tasks completed successfully in all three independent runs.

3.It would be helpful to report the wall-clock time for successful episodes of the open-weight models and specify the hardware configuration used to run them.

Reviewer_LrpeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AeroCopilotBench and its executable cockpit environment, ACOE, for evaluating LLM agents on aircraft emergency procedures. The ACOE defines state-transition rules, task goals, and trajectory-level safety constraints based on aircraft-specific Pilot's Operating Handbooks. The benchmark contains 73 tasks derived from 12 scenario templates across two aircraft types, which aim to evaluate task completion, safety compliance, execution discipline, and repeatability. The authors evaluate 12 models through repeated runs and analyze shortcomings in complete procedure execution, consistency across runs, and aircraft-specific emergency response.

Critical strengths
  1. The paper proposes an important application setting and a useful evaluation resource. Aircraft emergency procedures provide a meaningful setting for studying state-dependent decisions, procedural completion, and safety compliance. The executable environment enables evaluation of consequential tool actions rather than relying solely on textual answers.
  2. Auditable evaluation with substantial empirical coverage. Prerequisite-dependent transitions and trajectory-level constraints allow the benchmark to distinguish successful final outcomes from compliant execution.
  3. POH references and deterministic replay support inspection of individual judgments.
  4. Repeated evaluation of 12 models, together with trajectory-level failure analyses, provides a comprehensive evaluation.
Critical weaknesses
  1. The related-work discussion omits FLY-EVAL++, which already provides deterministic, constraint-based aviation safety evaluation. The paper fails to establish the methodological novelty beyond the proposed emergency-procedure setting.
  2. The broader contribution beyond an aviation-specific evaluation benchmark remains unclear. The failure analyses identify recurring problems but do not sufficiently distinguish missing aircraft-specific knowledge from difficulties in selecting and executing procedures. This ambiguity limits what the findings tell us about which capabilities need improvement and how to develop more reliable agents.
  3. The benchmark’s validation is insufficiently documented. Although the tasks are grounded in POHs and checked through reference solvers and manual review, the paper provides little detail about the reviewers’ aviation expertise, the review process, or how scoring judgments were verified. These omissions make it difficult to assess whether the formalized rules reliably capture the intended procedural requirements.
  4. The benchmark is not provided, preventing inspection of the implementation and reproduction for future research.
Questions for authors
  1. What evidence separates aircraft-specific knowledge gaps from procedure-selection and execution failures, and what does this reveal about how to improve agent reliability?
  2. What aviation expertise and review protocol were used to verify that the formalized rules and scoring judgments accurately reflect POH requirements?
  3. What methodological advances does AeroCopilotBench offer over FLY-EVAL++ beyond the emergency-procedure setting?
  4. Beyond detecting failures through safety-gated evaluation, does the paper identify any concrete findings improve models’ procedural reliability and safety?
1 comment

Thanks for your thoughtful responses. The responses address most of my concerns, although the benchmark is still not available. I hope the authors will release the environment, tasks, and evaluation code with the camera-ready version. I will raise my rating to 3.

Reviewer_3NySOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AeroCopilotBench evaluates LLM agents on aircraft-specific emergency procedures in a stateful, executable cockpit environment. The benchmark contains 73 tasks derived from 12 scenario templates for the Cessna 172S and Piper PA-44-180. An episode succeeds only when all terminal goals are achieved without any safety violation throughout the trajectory.

Across 2,628 runs involving 12 models, the highest safety-gated success rate was 72.6%. Most failures reached the critical goals but missed other required final conditions. The authors also observed inconsistent execution across repeated runs and apparently reasonable actions that conflicted with aircraft-specific manuals. The results suggest that aviation knowledge alone does not guarantee reliable, complete, and safe procedure execution.

Critical strengths
  1. The paper moves beyond static aviation question answering by introducing an executable cockpit in which actions produce outcomes through prerequisite-dependent state transitions.

2.The evaluation conducts a diagnostic analysis that combines task completion, safety compliance, execution discipline, and repeatability, then uses trajectories to examine why models fail.

Critical weaknesses
  1. The evaluation requires models to recall emergency procedures without manual or checklist retrieval. Without a retrieval-assisted comparison, it is difficult to assess how much of the observed failure is due to missing aircraft-specific information and how much reflects limitations in executing a procedure even when it is available.

  2. Each task starts from a predefined abnormal state and follows deterministic, task-specified transition rules; the Pilot Flying does not intervene after the initial instruction. This provides limited assessment of how agents respond to evolving situations or new crew instructions. Consequently, the benchmark’s conclusions are more directly applicable to execution under predefined conditions than to broader emergency decision-making.

Questions for authors
  1. How much of the observed failure is attributable to missing aircraft-specific knowledge versus difficulty executing a procedure when it is available? Could the authors include a comparison in which models can retrieve the relevant POH procedures or checklists?

  2. How might the benchmark be extended to assess responses to evolving situations or new crew instructions? How do the authors expect these interactions to affect the reported findings?

Reviewer_7SZjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AeroCopilotBench, a benchmark to evaluate LLM agents' capability and aviant knowledge in aircraft operation scenarios. The benchmark comprises 12 scenario templates and 73 tasks across two aircraft types, evaluating multiple metrics beyond the overall SR. Results on mainstream LLMs show that: 1) agents tend to partially solve the tasks; 2) sometimes do not follow a consistent procedure across multiple trials; and 3) agents do not follow their reasoning faithfully.

Critical strengths

Domain-specific benchmarks are valuable and requires substantial efforts to collect and annotate high-quality data. The logic of this paper is generally clear, making it easy to follow. The experiments cover various models and metrics, and further analysis reveals more than trajectory-wise metrics.

Critical weaknesses
  1. The contribution over existing works are not well discussed. Specifically, how would you clarify the novelty against PilotBench (and other aviant LLM evaluation works)? What new capabilities of agents are evaluated? Which new patterns and issues are found? Without clear explanations, it is hard to evaluate the novelty of this work.

  2. The benchmark creation process and LLM inputs are not well described:

In Section 4.1 there is no definition of any reference solver or trajectory. However, in Section 4.3 the authors mentioned that "a reference solver must receive 1 without any safety violations", and in Appendix D.4 and E.1 the authors mentioned "reference steps (trajectories)". How do you implement such reference solver and trajectories? L263: "Model-assisted checks focus on issues revealed during interaction." How do you identify such issues and filter accordingly? For manual review, how many candidate tasks are produced and how do humans revise the task?

  1. Lack of domain-specific knowledge and lack of agentic capabilities are not well decoupled. Therefore, it is not clear to me what is the root cause of agents' failure, and how well the conclusions about agent capabilities can transfer to other domains. Specifically:

The prompt says "Work the problem methodically and safely according to standard Pilot’s Operating Handbook procedures, which you must recall from your own aviation knowledge", which requires the agents to rely on its own parametric knowledge without external ground-truth information. The boundary between information given to agents or not is not clear. Do agents know the effect of each possible actions? To what extent do agents understand the constraints? How do they relate to the transition rules? These are not clear. Given all the findings, the authors do not provide a (prototypical) solution to verify that the findings provide actionable insights that can help in boosting LLM performances.

Questions for authors

See weaknesses above. If the authors can decently address the concerns, I am willing to consider raising my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.