- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
-
This paper introduces formal300, a benchmark of 300 FAA approach charts for reconstructing missed-approach procedures as ordered, canonical structures, with references derived from official FAA navigation records.
-
This paper shows that field-level accuracy substantially overstates structural understanding: chart-blind baselines score well on individual fields, while even the strongest systems rarely recover complete procedures.
-
This paper validates the reliability and recoverability of its references through an external blind re-audit and a blind calibration with trained human readers.
Critical strengths
-
Evaluation design that exposes misleading metrics. Through multi-level scoring and chart-blind baselines, the paper shows that high field-level accuracy can coexist with near-total failure in complete procedure reconstruction. This insight applies broadly to structured document extraction.
-
The references come from official FAA records and are validated by recoverability audits, an external blind re-audit, and trained-reader calibration. This ensures that low model scores reflect genuine model limitations rather than label errors.
Critical weaknesses
-
Unclear novelty relative to prior work. The related work section does not clearly establish what formal300 evaluates beyond existing structured-extraction benchmarks and diagram-to-structure tasks. Its distinction rests mainly on the FAA domain, without arguing why these charts pose a distinct challenge. As a result, the contribution risks reading as a domain-specific dataset rather than a broader advance in document understanding.
-
Limited evaluation of fine-tuned models. Despite providing a training set, the paper fine-tunes only one small model (Qwen2-VL-2B), which already far outperforms all untuned models. Without larger or newer fine-tuned models, it remains unclear whether the observed reconstruction gap reflects a fundamental model limitation or limited training effort, weakening the paper’s central claim about task difficulty.
Questions for authors
-
What capability does formal300 evaluate that existing structured-extraction benchmarks and diagram-to-structure tasks do not? Is there evidence that the gap between field-level and full-procedure accuracy is specific to FAA charts rather than common to these settings?
-
Why was fine-tuning limited to a single small model (Qwen2-VL-2B, one epoch, 4-bit LoRA)? Could the authors report results for larger or newer open models?
-
What are the qualifications and experience of the reference curators, external re-auditors and blind readers, and are any of them authors? How were they recruited and compensated?
2 comments
Contributions and Stronger Baseline Results
We apologize for the delayed response.
Thank you for recognizing the reference validation and the value of multi-level evaluation. We address the contribution, the stronger fine-tuned baseline, and the human participants below.
1. Contribution and relationship to prior work
Engineering drawing research is constrained by a shortage of publicly available real-world data: proprietary datasets impede comparisons between methods, while simple synthetic drawings do not adequately represent real-world complexity [1]. Uneven dataset coverage also directs research toward tasks with established benchmarks [2]. formal300 addresses this need by pairing 300 real aviation charts with official records of the same procedures, providing 964 canonical legs and a unified evaluation protocol.
On these materials, systems must identify procedural operations and their order from text, symbols, and their relationships, and assign fixes, courses, altitudes, and holding parameters to the appropriate legs. For example, the same fix can serve as both a direct-to endpoint and a holding location. The paired official procedure references provide a common basis for evaluating whether this information has been organized into complete procedures. formal300 thus extends the real-world domain data and task coverage available for existing research on structured extraction from technical diagrams.
2. Is the local-to-global accuracy gap specific to aviation?
Thank you for raising this distinction. Our experiments do not include a cross-domain comparison testing whether this gap is specific to aviation, and we therefore do not interpret it as an aviation-specific phenomenon. The paper's dataset and benchmark contribution does not depend on the gap being unique to aviation.
3. A stronger fine-tuned baseline
Following your suggestion, we have added fine-tuning results for Qwen3-VL-8B-Instruct, using 500 training charts and 100 development charts, evaluated on the same 200 test charts.
| Fine-tuned configuration | Atom accuracy | Exact procedures |
|---|---|---|
| Qwen2-VL-2B (original) | 75.20% | 8/200 (4%) |
| Qwen3-VL-8B-Instruct (new) | 64.88% | 24/200 (12%) |
The stronger configuration improves complete-procedure reconstruction from 4% to 12%, while leaving substantial room for improvement. The different rankings under atom accuracy and procedure exact match also show that the benchmark can distinguish the levels at which methods improve, rather than reducing their performance to a single aggregate score.
4. Human participants
All participants in reference preparation, external re-auditing, and blind reading were affiliated with the same aviation-focused institution, which is different from the authors' institution. None were authors of the paper. Participants were recruited through contact with the institution's aeronautical information department. Compensation was determined on the basis of an estimated task workload of approximately two days. The three reference curators had training in flight procedure design and rotated through a workflow in which one person prepared each chart and two checked it. The two external re-auditors were graduate students specializing in aeronautical information and had also received flight procedure design training. The two blind readers had received relevant training and had not participated in reference construction.
References
[1] Jamieson, L., Moreno-García, C. F., & Elyan, E. (2024). A review of deep learning methods for digitisation of complex documents and engineering diagrams. Artificial Intelligence Review, 57, 136. https://doi.org/10.1007/s10462-024-10779-2
[2] Pagidi, K. R., & Gardas, A. (2026). Vision foundation models for engineering document intelligence and manufacturing inspection: a survey. Artificial Intelligence Review, 59, 228. https://doi.org/10.1007/s10462-026-11622-6
As the discussion period is approaching the end and the authors do not give a response to my concerns, I will keep my rating as clear rejection.