acceptodds
Under review as a conference paper at ICLR 2027

formal300: Evaluating Local-to-Global Procedure Reconstruction from FAA Approach Charts

Abstract

Professional charts and machine records encode the same procedure at different representational levels: charts distribute intent across text, symbols, and spatial geometry, whereas machine records require ordered and normalized fields. We introduce formal300, a benchmark for recovering canonical missed-approach procedure structure from 300 FAA approach charts with 964 normalized legs. Rather than treating extraction as independent field prediction, formal300 evaluates a hierarchy from atom correctness to exact legs and exact procedures under a frozen canonicalization and scoring contract. A chart-blind majority prior attains 58.12% atom accuracy but recovers no exact leg or procedure, showing that correct individual scoring atoms need not form a correct ordered procedure. A layout-aware domain-rule pipeline uses OCR layout and public procedure rules to assemble ordered legs, while an auxiliary-trained 2B-parameter vision-language system predicts canonical structures from chart images. Their errors on individual scoring atoms overlap only partially, and both achieve exact recovery for only a small fraction of complete procedures. Reference recoverability is supported by a blind 200-atom re-audit and a separate blind trained-reader calibration yielding 59/60 exact submissions on 30 charts. formal300 provides an auditable, controlled testbed for studying how distributed professional-chart evidence is converted into canonical ordered structure.

Primary area:
datasets and benchmarks
Keywords:
Datasets and Benchmarks, Multimodal Document Understanding, Structured Information Extraction, Chart Understanding, Vision-Language Model Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_9N5qOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary
  • This paper introduces formal300, a benchmark of 300 FAA approach charts for reconstructing missed-approach procedures as ordered, canonical structures, with references derived from official FAA navigation records.

  • This paper shows that field-level accuracy substantially overstates structural understanding: chart-blind baselines score well on individual fields, while even the strongest systems rarely recover complete procedures.

  • This paper validates the reliability and recoverability of its references through an external blind re-audit and a blind calibration with trained human readers.

Critical strengths
  • Evaluation design that exposes misleading metrics. Through multi-level scoring and chart-blind baselines, the paper shows that high field-level accuracy can coexist with near-total failure in complete procedure reconstruction. This insight applies broadly to structured document extraction.

  • The references come from official FAA records and are validated by recoverability audits, an external blind re-audit, and trained-reader calibration. This ensures that low model scores reflect genuine model limitations rather than label errors.

Critical weaknesses
  • Unclear novelty relative to prior work. The related work section does not clearly establish what formal300 evaluates beyond existing structured-extraction benchmarks and diagram-to-structure tasks. Its distinction rests mainly on the FAA domain, without arguing why these charts pose a distinct challenge. As a result, the contribution risks reading as a domain-specific dataset rather than a broader advance in document understanding.

  • Limited evaluation of fine-tuned models. Despite providing a training set, the paper fine-tunes only one small model (Qwen2-VL-2B), which already far outperforms all untuned models. Without larger or newer fine-tuned models, it remains unclear whether the observed reconstruction gap reflects a fundamental model limitation or limited training effort, weakening the paper’s central claim about task difficulty.

Questions for authors
  1. What capability does formal300 evaluate that existing structured-extraction benchmarks and diagram-to-structure tasks do not? Is there evidence that the gap between field-level and full-procedure accuracy is specific to FAA charts rather than common to these settings?

  2. Why was fine-tuning limited to a single small model (Qwen2-VL-2B, one epoch, 4-bit LoRA)? Could the authors report results for larger or newer open models?

  3. What are the qualifications and experience of the reference curators, external re-auditors and blind readers, and are any of them authors? How were they recruited and compensated?

2 comments

As the discussion period is approaching the end and the authors do not give a response to my concerns, I will keep my rating as clear rejection.

AuthorsOpenReview ↗

Contributions and Stronger Baseline Results

We apologize for the delayed response.

Thank you for recognizing the reference validation and the value of multi-level evaluation. We address the contribution, the stronger fine-tuned baseline, and the human participants below.

1. Contribution and relationship to prior work

Engineering drawing research is constrained by a shortage of publicly available real-world data: proprietary datasets impede comparisons between methods, while simple synthetic drawings do not adequately represent real-world complexity [1]. Uneven dataset coverage also directs research toward tasks with established benchmarks [2]. formal300 addresses this need by pairing 300 real aviation charts with official records of the same procedures, providing 964 canonical legs and a unified evaluation protocol.

On these materials, systems must identify procedural operations and their order from text, symbols, and their relationships, and assign fixes, courses, altitudes, and holding parameters to the appropriate legs. For example, the same fix can serve as both a direct-to endpoint and a holding location. The paired official procedure references provide a common basis for evaluating whether this information has been organized into complete procedures. formal300 thus extends the real-world domain data and task coverage available for existing research on structured extraction from technical diagrams.

2. Is the local-to-global accuracy gap specific to aviation?

Thank you for raising this distinction. Our experiments do not include a cross-domain comparison testing whether this gap is specific to aviation, and we therefore do not interpret it as an aviation-specific phenomenon. The paper's dataset and benchmark contribution does not depend on the gap being unique to aviation.

3. A stronger fine-tuned baseline

Following your suggestion, we have added fine-tuning results for Qwen3-VL-8B-Instruct, using 500 training charts and 100 development charts, evaluated on the same 200 test charts.

Fine-tuned configurationAtom accuracyExact procedures
Qwen2-VL-2B (original)75.20%8/200 (4%)
Qwen3-VL-8B-Instruct (new)64.88%24/200 (12%)

The stronger configuration improves complete-procedure reconstruction from 4% to 12%, while leaving substantial room for improvement. The different rankings under atom accuracy and procedure exact match also show that the benchmark can distinguish the levels at which methods improve, rather than reducing their performance to a single aggregate score.

4. Human participants

All participants in reference preparation, external re-auditing, and blind reading were affiliated with the same aviation-focused institution, which is different from the authors' institution. None were authors of the paper. Participants were recruited through contact with the institution's aeronautical information department. Compensation was determined on the basis of an estimated task workload of approximately two days. The three reference curators had training in flight procedure design and rotated through a workflow in which one person prepared each chart and two checked it. The two external re-auditors were graduate students specializing in aeronautical information and had also received flight procedure design training. The two blind readers had received relevant training and had not participated in reference construction.

References

[1] Jamieson, L., Moreno-García, C. F., & Elyan, E. (2024). A review of deep learning methods for digitisation of complex documents and engineering diagrams. Artificial Intelligence Review, 57, 136. https://doi.org/10.1007/s10462-024-10779-2

[2] Pagidi, K. R., & Gardas, A. (2026). Vision foundation models for engineering document intelligence and manufacturing inspection: a survey. Artificial Intelligence Review, 59, 228. https://doi.org/10.1007/s10462-026-11622-6

Reviewer_HXMdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents formal300 and evaluates reconstruction of ordered missed-approach procedures from FAA charts into canonical JSON. It contains 300 charts and 964 normalized legs, measures atom, leg, and procedure correctness separately, and studies rule-based, vision-language, and auxiliary-trained extraction pipelines.

Critical strengths

I am glad to see the paper distinguish recognizing individual fields from reconstructing a complete procedure. In Table 3, the chart-blind majority prior reaches 58.12% atom accuracy without recovering a single exact leg or procedure, showing why field accuracy alone can be misleading. An executable interface that also checks ordering, field assignment, and canonical structure is a useful contribution.

I appreciate the effort to address a central risk of machine-record-derived references: requiring information unavailable from the chart. The external blind re-audit agrees with the reference on 197/200 atoms before adjudication, while independent trained readers recover 59/60 complete procedure submissions on 30 charts.

Critical weaknesses

I have presented the weaknesses together with the relevant details to make it easier for the authors to address them during the rebuttal period. If my concerns are adequately addressed, I would be willing to raise my score.

The main results still make it difficult to distinguish reconstruction failure from inadequate adaptation to the canonical interface. A1+ has explicit domain grammar, legal defaults, and geometry rules; D1-aux500 has task supervision and an output adapter; the general-model protocols receive different inputs and prompts. The paper acknowledges that these are not controlled causal comparisons, which I appreciate, but a stronger interface-matched general-model reference is still needed.

Of the 4,052 scored atoms, 1,855—about 45.8%—are convention-inferable or legality-constrained rather than directly chart-supported. The paper explicitly acknowledges this, but the main results still combine them with the 2,197 chart-supported atoms, obscuring where A1+ and D1 obtain their gains. The distinction is especially important for turn, where only 52/642 atoms are classified as directly chart-supported. Since the audit categories already exist, please report both subsets for every main method, with field-level macro averages within each, while retaining procedure exact match as the endpoint. This would distinguish legal completion from chart evidence recovery and clarify the benefit of explicit domain rules.

The formal200 set contains 157 three-leg charts and 141 holding charts. D1 reaches 84.75% atom accuracy on three-leg cases but 48.55% on non-three-leg cases, and the harder slices recover some exact legs but no complete procedures. I appreciate the airport-overlap analysis, including stronger performance on the airport-disjoint subset, so I would not attribute the results simply to airport memorization. However, airport separation and exact-procedure deduplication do not establish generalization across abstract leg structures. Please report frequencies and macro-averaged results by path-terminator sequence or another interpretable structural template, with a slice containing combinations absent from training. This would clarify whether auxiliary training mainly teaches common procedure patterns or supports new structural combinations.

I agree that complete procedures deserve more attention than isolated fields, and the authors correctly state that the exact-match gap does not by itself establish binding failures. Nevertheless, that gap remains central to the paper, while position-wise alignment can propagate one missing or incorrectly segmented leg into several subsequent errors, and strict matching naturally compounds local mistakes.

Minor:A compact field-legality table would help clarify BOTH, not_applicable, unknown, direction conventions, and display equivalence for course and holding fields. Please also mark which rules are actually supplied in each model prompt.

Questions for authors

In the later matched C1 channel, one example raises atom accuracy from 39.88% to 68.21%, illustrating substantial protocol sensitivity. Please provide the main models with the same legal-value rules, canonicalization instructions, and reference-blind formatting policy, and report schema validity separately from structural correctness.

1 comment
AuthorsOpenReview ↗

A Strong Baseline and a Closer Look at What Missed-Approach Reconstruction Scores Measure

Thank you for your constructive suggestions.

1. With explicit instructions, Kimi k3.1-pro recovered 89/200 complete procedures (44.5%). After inspecting early test outputs, we clarified the procedure scope. The final protocol used full charts, public rules and three development examples across all 200 charts. Test answers were never supplied. Of 200 outputs, 199 were schema-valid as returned. Removing one extra metadata key without consulting the reference or changing procedure content made all 200 valid.

The 8B results confirm prompt sensitivity. On the same checkpoint and charts, revised rules without examples gave 196/200 schema-valid raw outputs and 2/200 exact procedures, versus 24/200 exact with the original prompt. Both new prompts specified BOTH for unrestricted CF/DF turns, distinct inapplicable/unobservable/unknown states, no independent DF course, radial versus travel direction, and holding inbound course. Specified angles round to whole degrees (reciprocals differ). Published hold time/distance takes precedence. Ordinary timed holds otherwise default to 1 min at or below 14,000 ft holding altitude and 1.5 min above. The original tuned prompts allowed BOTH. The text-input and both Claude prompts allowed only LEFT/RIGHT. Both new runs used the same reference-blind formatter. Other models kept their original prompts.

2. Rule-system and tuned-model gains are chiefly in chart-supported targets. Relative to the chart-blind prior, layout-aware rules gain 1,151 chart targets but only 51 convention targets; tuned Qwen2-VL-2B (a 2B vision-language model) gains 797 chart targets while losing 105 convention targets. Legal completion alone therefore cannot explain these advantages.

All comparisons use 4,011 targets: 2,191 chart-supported and 1,820 convention/legality targets, excluding 41 TF (track-to-fix) course fields whose inclusion convention was underspecified. The 200 charts and 642 legs are unchanged. Entries below are correct targets / field-macro accuracy (%), equally weighting six chart and five convention field categories; leg count enters chart totals only.

MethodChartConventionExact procedures /200
Chart-blind majority prior755/16.191567/79.920
Rules on recognized text387/13.23770/38.020
Layout-aware rules1906/80.911618/83.7724
Language model on recognized text389/23.44532/24.470
Claude Sonnet 4.5 (general model), chart605/32.53903/38.340
Claude Sonnet 4.5, chart + recognized text617/32.32872/43.690
Tuned Qwen2-VL-2B1552/59.091462/75.998
Tuned Qwen3-VL-8B (8B vision-language model), original prompt1386/64.171208/63.9624
Tuned Qwen3-VL-8B, revised rules1110/51.12990/50.822
Kimi k3.1-pro, rules + examples1969/87.581679/90.9789

3. All 2B successes fall in the dominant template. The 8B model also succeeds on two other familiar templates. With their original prompts, all eight 2B successes and 17 of the 24 8B successes are on the 149 climb-direct-hold charts. The other seven 8B successes span two of the eight remaining templates (51 charts). Giving each template equal weight yields 0.60% and 6.01% exact procedures for 2B and 8B.

The split tests recovery of new chart instances within familiar structures. Even a known sequence leaves the fix, altitude, holding parameters and their leg assignments to be recovered from each chart. All nine test sequences occur among the 22 training sequences. Testing unseen combinations would require a separate structural holdout.

SequenceTrain/testExact 2B/8B
CA-DF-HM188/1498/17
CA-DF-TF-HM77/340/6
CA-CF-HM21/40/0
CA-VI-CF-HM76/40/1
CA-DF-TF-TF-HM54/30/0
VI-CF-HM8/30/0
CA-DF1/10/0
CA-DF-TF1/10/0
CF-HM40/10/0
Other 13 sequences34/0—

CA: course to altitude; DF: direct to fix; TF: track to fix; CF: course to fix; VI: heading to intercept; HM: hold to manual termination.

4. Evaluation-time alignment restores credit while preserving autonomous structure recovery. For the same Kimi outputs, order-preserving alignment raises exact legs from 471 to 498, retaining missing/extra-leg penalties. This confirms your concern: 27 correct legs lost credit through misalignment. By matching existing predictions to the reference after generation, we still require the model to determine leg count, types and order. Supplying each test chart's correct skeleton instead would give away this part of the answer and turn structure recovery into filling a given skeleton.

A failed exact procedure need not imply widespread content failure: 70 of Kimi's 111 unsuccessful procedures have only one scored-field mismatch, confirming your concern about compounded local errors. The endpoint remains 89/200, requiring all necessary legs and attributes to be correct. Aligned scores and mismatch counts describe partial recovery; the score gap alone is not evidence of an additional failure mechanism.

Reviewer_JnHZOpenReview ↗
Rating
1
Confidence
2(−1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Summary

The paper introduces formal300, a benchmark for reconstructing canonical missed-approach procedure structure from FAA approach charts. It contains 300 charts (174 airports, 964 normalized legs) with a 50/50/200 airport-disjoint split. References are derived from evaluation-side CIFP/ARINC 424 records and projected into an ordered leg-level JSON with six scored fields per leg. Outputs are evaluated at three levels (atom accuracy, exact leg, exact procedure) by a frozen scorer. Reference validity is supported by an author audit of all 4,052 test atoms (2,197 chart-supported, 1,855 convention-inferable), a blind external re-audit (197/200 atoms agree), and a blind trained-reader calibration (59/60 exact procedures on 30 charts).

Validation: On formal200, the methods compared are:

  • chart-blind priors;
  • OCR+rules, OCR+LLM, and zero-shot VLM protocols (Claude Sonnet 4.5);
  • a QLoRA-tuned Qwen2-VL-2B trained on a separate 500-chart auxiliary set (D1-aux500);
  • a hand-built layout-aware rule pipeline (A1+).

A chart-blind majority prior reaches 58.12% atom accuracy with zero exact legs, and all zero-shot VLM rows score below this prior. The best systems are A1+ (87.81% atoms, 24/200 exact procedures) and D1-aux500 (75.20%, 8/200). Their error patterns are only partially overlapping. Additional analyses cover in-context examples, training seeds, cross-cycle probes, and source-view ablations.

Critical strengths

None

Critical weaknesses

1. The central finding largely follows from the benchmark design, and the contribution is too narrow for ICLR.

The headline "local-to-global reconstruction gap" is, in the end, the observation that high per-field accuracy does not imply exactly correct legs or procedures under strict matching. I don't think this tells us much about models. Exact match over ordered six-field legs compounds per-field errors by construction. More importantly, by the authors' own audit (Table 6), 1,855 of the 4,052 scored atoms (46%) are convention-inferable or legality-constrained rather than read from the chart, and this rises to 590 of 642 for the turn field. A chart-blind prior scoring 58.12% atoms and zero exact legs is therefore close to a direct consequence of the scoring contract, and the paper itself acknowledges the gap "does not by itself isolate binding-specific failures." Beyond this, the task is a single sub-procedure (FAA missed approaches) with a 200-chart test set whose references are released with the paper. The broader "professional charts vs. machine records" framing in the introduction is never tested outside this one setting. I appreciate the care in the annotation, but I struggle to see what the ICLR community learns from it that would transfer beyond this domain.

2. The baselines are too weak to support claims about task difficulty.

The only frontier model in the main comparison is Claude Sonnet 4.5. GPT-5.4 appears only in a small few-shot probe, and no current strong open VLM (e.g., Qwen3-VL-class models) is evaluated. The trained system is Qwen2-VL-2B with QLoRA, capped at ~0.5MP input, which seems low for dense approach charts; resolution is not discussed. Every zero-shot VLM row falls below the chart-blind prior (C1 37.69%, C4 37.19% vs. 58.12%). That looks more like an output-interface or prompting problem than a property of the task, and the jump to 68.21% with a single in-context example supports this reading. The strongest system (A1+) is a hand-engineered rule pipeline, and the authors state that rows in Table 3 are "descriptive protocol comparisons rather than controlled causal ablations." As a result, the paper establishes neither how hard the task is for current models nor what drives the remaining errors, which is what I would expect a benchmark paper to deliver.

Formatting. The submission deviates from the official ICLR template: it uses a Latin Modern typeface instead of Times, and section and abstract headings are not in the template's style. This does not appear to gain extra space, so I leave it to the AC whether any action is needed.

Questions for authors
  1. Chart-supported subset. Could you report atom, leg, and procedure exact match restricted to the 2,197 chart-supported atoms, i.e., excluding the convention-inferable ones? This would show whether the "local-to-global gap" persists once convention completion is factored out, or whether it is mainly an artifact of the scoring contract.

  2. Stronger baselines. Why are current strong VLMs (e.g., GPT-5.x, Gemini 3.x, or Qwen3-VL-class open models) not evaluated in the main comparison, and why is the trained model limited to Qwen2-VL-2B at ~0.5MP input? Given that every zero-shot VLM row scores below the chart-blind prior, can you rule out that this reflects the output interface or input resolution rather than task difficulty?

  3. Transfer beyond FAA missed approaches. Is there any evidence that the findings, or the benchmark design, transfer to other procedure types (e.g., SIDs/STARs) or to other chart systems? Without it, it is hard to see what the broader framing in the introduction rests on.

2 comments
AuthorsOpenReview ↗

The contribution and evaluation of formal300

Response to Reviewer JnHZ on the contribution and evaluation of formal300

Thank you for your comments on the contribution, the interpretation of the evaluation, and the reference for task difficulty. We address these three issues below.

1. What does formal300 contribute, and why does it matter beyond missed approaches?

Engineering drawing research has long been constrained by a shortage of publicly available real-world data: proprietary datasets impede method comparisons, while simple synthetic drawings do not adequately represent real-world complexity [1]. Uneven dataset coverage also directs research toward tasks with established benchmarks [2]. formal300 addresses this need by providing 300 real aviation charts paired with official records of the same procedures, 964 canonical legs, and a unified evaluation protocol, allowing structured-extraction methods to be trained and compared on new real-world domain materials.

The broader framing in the introduction concerns a shared problem: reconstructing ordered structure from information distributed across text, symbols, and spatial relations. Missed approaches provide a concrete instance through operation segmentation, cross-region attribute assignment, and sequence reconstruction. These paired materials therefore enable researchers developing structured-extraction methods beyond aviation to study the same problems on real technical diagrams.

2. When only chart-supported items are scored, does the gap between field accuracy and whole-procedure accuracy persist?

Removing convention-inferable items raises procedure accuracy for both methods. Even when scored only on chart-supported targets, both methods recover individual fields more often than they recover all the fields and legs of a procedure in order.

MethodExact procedures, including conventionsChart-supported field accuracyLegs with all chart-supported targets correctProcedures with all chart-supported targets correct
Existing layout-aware rule configuration12%86.99%72.59%37%
Kimi K3.1 Pro44.5%89.87%77.88%51.5%

Both methods are evaluated on the same 200 charts and 642 legs, with 41 course fields for TF (track-to-fix) legs excluded throughout. This leaves 4,011 scored items, including 2,191 chart-supported items. The chart-only evaluation retains leg-count, order, and correspondence requirements.

The chart-only evaluation identifies exactly which failures can be repaired by correcting conventions alone. A procedure that passes the chart-only check but fails the full check already has the required chart content in the right order. For the rule configuration, more procedures fall into this category, which explains why its deficit narrows when conventions are removed. Even if every convention entry were corrected, its full-procedure accuracy with the same chart predictions could reach only its chart-only score, which is already below Kimi’s observed full-procedure accuracy. Closing that gap requires recovering the chart content of additional procedures in full.

3. How difficult is the task for current models, and what explains their remaining errors?

Given the full chart, explicit output rules and three development examples, Moonshot AI’s vision-language model Kimi K3.1 Pro reconstructed nearly half the procedures exactly, outperforming the existing chart-specific rule system on the same charts. The fine-tuned 8-billion-parameter Qwen3-VL-8B-Instruct model also recovered more complete procedures with its original prompt than the original 2B model. In a separate comparison under the original scoring protocol, approximately doubling the input pixel count increased exact recoveries with weights and prompt held fixed.

Pairing each chart with its official record lets us pinpoint a correction that would turn a failed reconstruction into an exact one. On KAAF_R24, Kimi produced a valid response and identified the correct missed-approach holding fix. Its only error was to assign that fix a course printed at another holding pattern on the same chart. Reading the inbound course from the holding pattern associated with the named fix would correct this single error and recover the entire procedure.

References

[1] Jamieson, L., Moreno-García, C. F., & Elyan, E. (2024). A review of deep learning methods for digitisation of complex documents and engineering diagrams. Artificial Intelligence Review, 57, 136. https://doi.org/10.1007/s10462-024-10779-2

[2] Pagidi, K. R., & Gardas, A. (2026). Vision foundation models for engineering document intelligence and manufacturing inspection: a survey. Artificial Intelligence Review, 59, 228. https://doi.org/10.1007/s10462-026-11622-6

Thank you for the response and the new results. However, the rebuttal confirms my main concerns: once a stronger VLM (Kimi K3.1 Pro) is given clarified rules and 3-shot examples, exact procedure accuracy jumps from 0% (Claude in the paper) and 12% (A1+) to 44.5%, and reaches 51.5% when convention-inferable fields are excluded (while the original Claude prompt did not even allow BOTH for turns). You also agree that the drop from field accuracy to procedure exact match comes from error compounding under strict matching rather than a new failure mechanism, and the benchmark is still limited to 200 test charts of a single FAA sub-procedure. Therefore, I will maintain my score at 1.

Reviewer_3qreOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces formal300, a benchmark for reconstructing the ordered structure of missed-approach procedures from 300 FAA approach charts (AIRAC cycle 2604), covering 174 airports and 964 normalized missed-approach legs. The ground-truth annotations are derived by projecting CIFP/ARINC 424 records into canonical JSON. Each leg is evaluated based on six attributes, together with a separate leg-number field. The evaluation is conducted at three levels: atom accuracy, leg exact match, and procedure exact match. On the 200-chart test set, a chart-blind majority baseline achieves 58.12% atom accuracy but fails to reconstruct even a single leg completely. The authors' layout-aware rule-based pipeline, A1+, achieves 87.81% atom accuracy, with 344 correctly reconstructed legs across 24 complete procedures. Meanwhile, Qwen2-VL-2B fine-tuned on 500 auxiliary charts (D1) achieves 75.20% atom accuracy, with 198 correctly reconstructed legs across 8 complete procedures. General-purpose VLMs and LLMs achieve atom accuracy ranging from 23% to 38%. The paper also provides a ground-truth recoverability audit, in which an external blind review successfully verifies 197 out of 200 charts, as well as a blind evaluation by two independent human readers, who achieve 59 fully correct reconstructions out of 60 chart readings.

Critical strengths
  1. The problem formulation is novel and practically meaningful. The task focuses on reconstructing ordered structures that align with machine-interpretable procedural semantics, rather than performing localized question answering or generic field extraction. The data sources are publicly available, and the correspondence with ARINC 424/CIFP records is clearly established.
  2. The evaluation protocol is carefully designed and rigorously implemented. Both the scoring script and the normalization contract are frozen. Parsing failures and missing legs are consistently counted in the full denominator; unknown values are not granted wildcard credit; the chart-blind prior is explicitly reported; the input boundaries are clearly specified in Table 1; chart-level bootstrap confidence intervals are provided; and the supplementary materials support offline reproduction of the evaluation. These practices represent a relatively high standard for benchmark papers.
  3. The recoverability of the ground truth is supported by multiple layers of evidence. All atoms are categorized as either directly readable from the charts or inferable from established conventions, with no CIFP-only atoms. The authors further provide an external blind audit and an independent human-reader blind evaluation. The human-reader evaluation achieves 98% procedure-level exact accuracy, substantially exceeding the best system's performance of 12%, indicating that the benchmark retains considerable room for improvement.
  4. The authors are transparent about the limitations of their work. These include the non-causal nature of the comparisons, the selective nature of the dataset, and the fact that the benchmark is not intended for operational use.
  5. The error complementarity analysis provides useful insights. The errors made by A1+ and D1 are not subsumed by one another, suggesting that their respective strengths and weaknesses are complementary. This finding offers valuable insights for developing future model-fusion approaches.
Critical weaknesses

The manuscript has a noticeable formatting issue, the main-text font does not comply with the official template requirements. Although the manuscript includes the official header, footer, and line numbering, the font used in the main text does not conform to the specifications of the official template. To be fair, Latin Modern is wider than Times, so this deviation does not provide the authors with any advantage in terms of page limits. Nevertheless, it still constitutes a modification of the official formatting template. Whether this issue warrants further action under the ICLR 2027 author guidelines is ultimately at the discretion of the AC/PC. My overall assessment is based on the scientific contributions of the work.

  1. The "local-to-global gap" largely results from the progressive compounding of strict matching requirements and does not, in itself, constitute a novel finding. The chart-blind prior achieving zero fully correct legs is almost inevitable by construction, since waypoint names, altitudes, and headings vary across charts. Under a rough independence assumption, an atom accuracy of 87.81% for A1+ would be expected to yield approximately 46% leg exact accuracy and 13% procedure exact accuracy. The observed results are 53.6% and 12%, respectively. This suggests that the reported performance gap does not exceed what would be expected from the inherent compounding effects of exact matching. Furthermore, the paper does not provide an analysis that isolates field-to-leg binding errors from other types of errors. Another concern is that legs are aligned positionally during scoring. Consequently, a single extra or missing leg at the beginning of a procedure can shift the alignment of all subsequent legs, resulting in cascading mismatches.
  2. The primary metric conflates chart-reading ability with the ability to infer information from coding conventions. Of the 4,052 atoms in the benchmark, 1,855 rely on convention-based completion or inference (for example, 590 out of 642 atoms in the turn field). This distinction may explain several observations in the experimental results: the chart-blind prior achieves approximately 58% accuracy, general-purpose VLMs (37.69%) perform even worse than this prior, and one-shot prompting yields an improvement of approximately 28 percentage points. The authors have already classified each atom according to its source of information, yet they do not report the performance of different methods separately on these subsets. Such a breakdown is necessary to distinguish genuine chart-reading capabilities from the ability to infer missing information based on established coding conventions.
  3. The comparisons across different methods are not sufficiently fair. A1+ has access to FAA/ARINC rules and default values, but the paper does not clarify whether the prompts provided to the VLMs include the complete normalization contract. Furthermore, the strongest-performing system, A1+, and the ground-truth annotations and normalization contract were all developed by the same team, while A1+ was evaluated separately only after the original benchmark had been frozen. This raises concerns about potential methodological bias, as part of the benchmark may inadvertently measure the extent to which a system conforms to the authors' own conventions, rather than its generalizable chart-understanding capabilities.
  4. The baselines are too weak to adequately characterize the benchmark's difficulty and the remaining performance headroom. The learning-based baselines include only a single 2B-parameter model, with an input resolution of approximately 0.5 megapixels, which is likely insufficient to reliably resolve the small text in information-dense approach charts. Among the hosted models reported in the main results, only Claude Sonnet 4.5 is included. Furthermore, the paper does not specify which LLM backbone is used for the B-series baseline. These limitations make it difficult to determine whether the benchmark genuinely challenges the capabilities of current state-of-the-art models or whether the observed performance gaps are partly attributable to weak baseline configurations.
Questions for authors
  1. Comparison with the independence-based compositional expectation and alternative alignment strategies. Please compare the observed leg-exact and procedure-exact accuracy with the corresponding compositional expectations under the assumption of independent field-level errors. Could the authors also provide a position-insensitive evaluation as a complementary analysis (e.g., checking whether the value of each field appears in any predicted leg, regardless of its position) to quantify field-to-leg binding errors? Furthermore, how would the results change if sequence-based alignment were used instead of positional alignment?
  2. Disaggregated evaluation by atom category. Please report the performance of each method separately on the chart-supported subset (2,197 atoms) and the convention-inferable subset (1,855 atoms). In addition, please provide field-level results to clarify the strengths and weaknesses of different methods across individual attributes.
  3. Model input resolution and stronger learning-based baselines. What are the actual image resolutions provided to each model during evaluation? How does D1 perform when using higher-resolution inputs or tiled image inputs? Could the authors include SFT results for stronger open-source VLMs to provide more competitive learning-based baselines? Finally, which specific LLM backbone is used for the B-series models?
1 comment
AuthorsOpenReview ↗

Stronger Baselines and Disaggregated Evaluation for Procedure Reconstruction

Thank you for recognizing the significance of the task and the evaluation design. Our work provides real approach charts paired with official records for the same procedures, enabling research on reconstructing canonical missed-approach structures. Following your suggestions, we have added stronger models and disaggregated analyses to establish a more competitive performance reference and clarify how different target categories and evaluation requirements affect the results.

Stronger models substantially improve complete reconstruction. We have added K3.1 and supervised fine-tuning of Qwen3-VL-8B-Instruct, and completed a higher-resolution comparison for the original 2B model. K3.1 receives explicit task, field, and rule specifications, together with three development-set examples. The main results are:

MethodAtom accuracyExact legs /642Exact procedures /200
Original fine-tuned 2B model75.14%1988 (4%)
Enhanced rule-based system87.86%34924 (12%)
Newly fine-tuned 8B model64.67%25524 (12%)
K3.190.95%47189 (44.5%)

All rows use the same 200 test charts and the same supplementary evaluation scope of 4,011 targets. The accompanying supplementary tables specify this scope, the paired resolution comparison, and model configurations.

K3.1 exceeds the original rule-based reference at all three evaluation levels. We will update the main results accordingly, grounding the discussion of complete reconstruction in a more competitive configuration.

Convention-based completion affects exact-procedure results but does not explain all remaining mismatches. Following the paper's source categories, we report chart-supported targets separately from those requiring completion under coding conventions or subject to legality constraints:

MethodChart-supported atom accuracyConvention-inferable atom accuracyProcedures with all chart-supported targets correct
Enhanced rule-based system86.99%88.90%74/200
K3.189.87%92.25%103/200

For K3.1, excluding the completion requirements increases the number of fully correct procedures from 89 to 103, while 97 charts still fail to satisfy all chart-supported targets. Thus, completion requirements alone cannot explain the remaining mismatches. The same exclusion allows 50 additional procedures to pass for the rule-based system, compared with 14 for K3.1, showing that these requirements affect the two methods differently. Full source-category and per-field breakdowns for the main methods are provided in the supplementary tables.

Disaggregated diagnostics distinguish sources of lost credit in complete reconstruction. We agree that error compounding under strict matching is not, by itself, evidence of a new mechanism. The independence-based compositional reference quantifies the expected decline, while position-insensitive matching and order-preserving alignment examine the effects of misalignment; full results are provided in the supplementary tables. Order-preserving alignment increases K3.1's exact-leg count from 471 to 498, showing that positional scoring penalizes some content that has already been recovered. Alignment does not recover omitted content, and the exact-procedure count remains 89.

The source breakdown identifies which target categories account for lost credit, and alignment identifies positional penalties on recovered content. The original complementarity analysis of the rule-based and fine-tuned models further shows that the methods succeed on different cases. These diagnostics provide empirical insights beyond aggregate scores.

Thank you also for pointing out the formatting issue. We will correct the fonts and heading styles to comply with the official template.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.