acceptodds
Under review as a conference paper at ICLR 2027

AeroProc-Bench: Distinct Citation and Safety Failures Create a Deployment Dilemma in Aviation Maintenance LLMs

Abstract

Large language models deployed as maintenance assistants can cite manual references correctly while silently omitting mandated safety steps, a failure invisible to citation-based evaluation. We introduce AeroProc-Bench, a 4-dimensional benchmark built from 14,721 real aircraft fault-isolation items, and use it to reveal a deployment dilemma: citation accuracy and safety coverage are structurally independent per item () yet covary across methods (), so no training-free method optimizes both. The dilemma manifests as a ranking inversion at the frontier: the safest model achieves SEF 0.834 but scores only CEF 0.196, meaning a citation-only deployer would reject the safest available model. Supervised fine-tuning escapes the dilemma by dominating both axes, but is unavailable for frontier APIs. Single-metric evaluation therefore systematically misleads deployment in safety-critical domains; multi-dimensional, per-domain validation is essential. Code and scoring scripts are released in the supplementary material.

Primary area:
datasets and benchmarks
Keywords:
benchmark, LLM evaluation, hallucination, safety-critical AI, aviation maintenance, deployment dilemma

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_HXrPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces AeroProc-Bench, a four-dimensional benchmark built from 14,721 real aircraft fault-isolation items, mainly from Airbus A320 and A330 troubleshooting manuals. It highlights that an LLM can cite the correct source while still omitting critical safety steps, showing that citation accuracy alone is insufficient for evaluating safety-critical deployment.

Critical strengths
  1. Always appreciate seeing real-world, underrepresented engineering applications addressed in mainstream CS research conference. This work targets aviation maintenance, an important safety-critical domain, using a large benchmark derived from real troubleshooting procedures.
  2. Rather than evaluating only citation correctness, the paper examines citation failures and missing safety preconditions from multiple perspectives. This multidimensional evaluation is useful because it exposes failure modes that would be hidden by a single metric.
Critical weaknesses
  1. The paper contains several potentially overstated claims, and some of the experimental designs do not fully support the strength or generality of the conclusions drawn from them.
  2. At its core, this work can still be viewed as a domain-specific hallucination benchmark, an area that has already been extensively studied with many existing benchmarks and mitigation approaches. The paper therefore has difficulty clearly distinguishing its methodological novelty beyond applying multidimensional hallucination evaluation to the aviation domain.
Questions for authors

Most of the detailed intervention experiments are conducted on relatively small open-source models, while the frontier-model evaluation is more limited. Given that hallucination behavior can differ substantially in modern closed-source frontier LLMs, could the authors provide more comprehensive evaluation of these models, particularly across all four benchmark dimensions? From an LLMOps and deployment perspective, conclusions about safety-critical applications would be more convincing if they were validated on models that would realistically be considered for such deployments.

Reviewer_ebTBOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces AeroProc-Bench to evaluate citation omission, causal-branch confusion, cross-fault mis-attribution, and safety omission for procedural hallucination in aviation maintenance. AeroProc-Bench evaluates intra-document procedural accuracy, whether the model cites the correct manual references and includes mandated safety steps. The benchmark contains 14,721 items derived from maintenance manuals. Its main contributions are a domain-specific benchmark and empirical analysis of citation and safety failures.

Critical strengths
  • Jointly evaluating citation omission and safety-precondition omission in real aviation maintenance procedures is practically relevant.
  • The reported human-agreement checks for citation annotations and safety-coverage judgments provide some credibility to these specific labeling tasks.
  • This paper gives the study potential value as a starting point for analyzing omission errors in specialized procedural generation.
Critical weaknesses
  • The paper is hard to follow and the contribution is thus vague. For example, in the introduction section, there is no detailed research background, related works, scientific question, challenges, motivations, and benchmark construction process.
  • The whole method is ambiguous. In the method section (BENCHMARK CONSTRUCTION), there is only some benchmark statistic with no clear process showing how to construct the benchmark.
  • For the experiment part, important metric definitions and experimental details are overly compressed.
  • The figures also provide limited explanatory value: the pipeline remains high-level without providing detailed processes.
Questions for authors
  • The manuscript needs a clearer progression from problem definition through evaluation to conclusions. These presentation issues materially hinder assessment of the contribution and its supporting evidence.
Reviewer_SY9yOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This is a dataset-and-benchmark-track paper. It aims to evaluate LLM ability to cite the known procedure (just like RAG).

Critical strengths

This paper needs significant rework, as discussed in detail in the weaknesses section.

I have no positive points for this paper, except that the general direction is important from a safety perspective, as industrial assets such as Engines are critical. So I consider evaluating an LLM/AI Agent in this domain a critical need, and I appreciate the author for taking a first cut.

Critical weaknesses

The submitted paper currently reads like a technical report, mostly written by an LLM, and needs significant time to reach ICLR-level quality. It is submitted as a Dataset and Benchmark track paper and evaluated keeping the track requirements in mind.

  • I read the Abstract, and the term " maintenance assistants needs citations. Please read the literature and identify which related work or industry uses the work, and do the work,

  • Several uncommon terms, like citation accuracy and safety coverage, are used but not defined.

  • The abstract uses acronyms without defining them.

  • The citation is needed: leadership should not be a single metric but a multi-metric approach, and the domain has plenty of related search work.

  • It seems the paper was written using an LLM, which is fine, but it has long sentences that sometimes create a human long-context problem; they miss context.

  • Line 048: need references to the dataset.

  • Table 1 misses very important and relevant literature in the domain, such as a) IndustryBench, b) SOPBench, c) FailureSensorIQ 4) ITOFrmer 5) PHMForge, etc. Please update the table with all relevant recent work and compare and contrast. This also needs revision in Section 2; a laundry list of related work is not acceptable.

  • Figure 2 is not explained anywhere. I suggest the author review the papers above and learn how to adapt their methodology to describe how the dataset was generated, why it is challenging, and what innovations were needed.

  • Section 4.1: The paper should justify the model selection; we cannot pick an arbitrary set of models. Again, read how the benchmark paper selects the model for evaluation.

  • Sectio. 4.2 - I was expecting the author to put a pointer to the prompt, parameters, and an example so we can understand what type of questions are there. Without giving enough information, it is tough to get a better understanding og several charts that have been presented (Figure 2, Figure 3, etc)

  • I like Section 4.4 on cross-domain generalization, but it is not sufficient to draw any conclusions at this point.

  • References are the biggest issue of the paper. A good paper will prepare references with care, not with et.al. and all. Please use BibTeX (original) where all authors, etc are presented. Please look at a good published ICLR paper and how it formats its references (I mean everything about the style).

  • The Appendinx section is a bunch of diagrams and prompts, but the main paper must provide all the details.

  • No LLM was used to read the paper, write the review, or upload it anywhere. We printed the paper, read it in print, and drafted the review.

Questions for authors

I have no questions. The paper needs improvement that cannot be addressed by my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.