- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces AeroCopilotBench, a benchmark for evaluating how large language model agents execute aircraft emergency procedures in an executable cockpit. It covers the Cessna 172S and Piper PA-44-180 and systematically evaluates 12 models from different providers and model families. After receiving a task instruction from the Pilot Flying (PF), the models rely on their own aviation knowledge and feedback from the environment to carry out the procedures. The evaluation covers task completion, safety compliance, execution discipline, and repeatability, with safety-gated success rate as the primary metric. The paper also provides a relatively detailed analysis of failed executions. In my view, the benchmark contributes to research on large language model assistance in flight operations.
Critical strengths
1.To my knowledge, this study introduces the first benchmark for evaluating how large language model agents execute aircraft-specific emergency procedures in an interactive cockpit. It contributes to research on the use of large language models to assist flight operations. 2.The evaluation covers a range of widely used large language models, allowing their performance to be compared across model families.
Critical weaknesses
1.The paper does not clearly explain the significance of the proposed benchmark for the field or discuss its potential applications.
Questions for authors
1.The authors should discuss the research significance and potential implications of using large language models to carry out aircraft emergency procedures.
2.Why was the temperature set to 0.2 rather than 0 for all evaluated models? This choice may substantially affect the reported repeatability metric (pass³), which measures the proportion of tasks completed successfully in all three independent runs.
3.It would be helpful to report the wall-clock time for successful episodes of the open-weight models and specify the hardware configuration used to run them.
Thanks for your thoughtful responses. The responses address most of my concerns, although the benchmark is still not available. I hope the authors will release the environment, tasks, and evaluation code with the camera-ready version. I will raise my rating to 3.