- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces AeroProc-Bench, a four-dimensional benchmark built from 14,721 real aircraft fault-isolation items, mainly from Airbus A320 and A330 troubleshooting manuals. It highlights that an LLM can cite the correct source while still omitting critical safety steps, showing that citation accuracy alone is insufficient for evaluating safety-critical deployment.
Critical strengths
- Always appreciate seeing real-world, underrepresented engineering applications addressed in mainstream CS research conference. This work targets aviation maintenance, an important safety-critical domain, using a large benchmark derived from real troubleshooting procedures.
- Rather than evaluating only citation correctness, the paper examines citation failures and missing safety preconditions from multiple perspectives. This multidimensional evaluation is useful because it exposes failure modes that would be hidden by a single metric.
Critical weaknesses
- The paper contains several potentially overstated claims, and some of the experimental designs do not fully support the strength or generality of the conclusions drawn from them.
- At its core, this work can still be viewed as a domain-specific hallucination benchmark, an area that has already been extensively studied with many existing benchmarks and mitigation approaches. The paper therefore has difficulty clearly distinguishing its methodological novelty beyond applying multidimensional hallucination evaluation to the aviation domain.
Questions for authors
Most of the detailed intervention experiments are conducted on relatively small open-source models, while the frontier-model evaluation is more limited. Given that hallucination behavior can differ substantially in modern closed-source frontier LLMs, could the authors provide more comprehensive evaluation of these models, particularly across all four benchmark dimensions? From an LLMOps and deployment perspective, conclusions about safety-critical applications would be more convincing if they were validated on models that would realistically be considered for such deployments.