acceptodds
Under review as a conference paper at ICLR 2027

Vision-Language Models on Specification-Governed Instrument Approach Charts: Which Failures, and Which Supplied Information Changes Them

Abstract

Converting engineering drawings into structured data requires recovering every specification-defined instance and its attributes, and recent vision-language models are a candidate solution. On instrument approach charts, they omit instances or fill in wrong attributes. Scoring whole records as right or wrong cannot tell an unformed instance from a nonconforming field set or a misread value, nor which information to supply. From charting specifications and chart evidence, we define the instances and attributes each chart requires; this shared semantic reference scores outputs and yields the required instances (instance list), the fields to fill (field list), and reading rules. We evaluate Luna on 862 charts and Luna, Qwen, and K3 on a common 171-chart subset. Most strict failures are required instances that never form a record; with the instance list, matching is almost fully restored, nearly all newly correct records are previously unmatched targets recovered whole, and matched instances show almost no net change. Unformed instances account for 85.28% and 92.16% of Luna's and Qwen's strict failures; with the list supplied and identities still self-written, matching rises from 45.7%, 12.2%, and 56.7% to 98.8%, 73.9%, and 98.9%. The field list mainly aligns output fields with the scored reference fields; its effect on unsupplied values depends on response design. The reference makes each failure type separately measurable, turning an accuracy gain into an account of which information changed what.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, document understanding, instrument approach charts, structured information extraction, semantic completeness

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_zS3aOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents a specification-grounded reference and diagnostic methodology for structured extraction from FAA instrument approach charts. It evaluates Luna on 862 charts and compares three deployments on a common 171-chart subset, separating identity matching, field-set agreement, and value correctness. The target-level gain accounting and fixed-denominator field diagnostics provide useful insights beyond aggregate accuracy. Overall, I lean toward rejection. Although the diagnostic contribution is valuable, inconsistent control rates and an insufficiently explained scored-field policy leave the central interpretations unresolved, collection-time confounding further limits intervention attribution.

Critical strengths
  • The task contract clearly distinguishes identity matching, supplied-entry responses, and strict correctness over all targets.
  • The paper distinguishes retrospective analyses from registered hypotheses and reports the unsupported difficulty-scale hypothesis directly.
Critical weaknesses
  1. Inconsistency evaluation data. My main concern is the inconsistency between Luna’s control matching rates in Secs. 6.1 and 6.3 and Tables D3, D8, and D9. Free-generation matching is reported as 40.50%, and adding reading rules changes it by +2.33 points, implying 42.83%. The free-plus-reading-rules control used against L4-self instead has 45.71% matching. Both comparisons specify 171 charts and 2,741 targets, so rounding cannot explain the difference. It remains unclear whether these results use different saved controls, condition definitions, or scoring versions. This makes me concerned about the accuracy of the evaluation data involved in this manuscript.

  2. Unclear scored-field policy. In Sec. 5.2 and Appendix A.2, the course printed beside the AWONE→ROCCK segment is unannotated and is therefore penalized as an extra field, but the paper does not explain why it is excluded from scoring. Because the same reference generates both the scoring targets and the supplied information, this policy directly determines strict failures and field-list gains. Symmetric deletion measures the resulting score difference, but it does not show whether the scored-field set adequately represents the declared scope. Appendix B also leaves 7 of 251 missed-approach-point discrepancies unexplained. This makes me concerned that the reported gains partly reflect annotation choices rather than model behavior.

Questions for authors

Is the dominant failure a model failure or a task-specification failure? The base prompt leaves four identity boundaries unspecified (Appendix A.1). Stating them (C1) raises Luna’s matching from 40.50% to 59.03%, and 26.95% of unmatched targets already have a key-compatible output. How much of the 85.28% “unformed instances” is a convention mismatch rather than a recovery failure, and why does the main condition omit these conventions?

What outcome would have contradicted the gain accounting? Since the instance list supplies identities verbatim, higher matching is expected by construction. What matters is accuracy on newly matched targets: about 60% (873 of ~1,466) for Luna, versus 89% (1,105/1,242) for targets matched in both conditions. Does this gap not suggest these targets are also harder to read?

Reviewer_vnuTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper evaluates vision-language models (VLMs) on extracting structured information from FAA instrument approach charts under conditions that vary the information provided to the models. The authors annotate 862 charts and evaluate GPT-5.6-Luna, Qwen3.5-9B, and K3. Unmatched records account for most reported errors. Providing record identities or specifying the required fields substantially improves the reported accuracy, with gains primarily attributable to record matching and field selection.

Critical strengths
  • Detailed evaluation and diagnostic analysis.

Experiments with Luna, Qwen3.5-9B, and K3 across multiple conditions distinguish record matching, field selection, and value errors, showing that higher aggregate accuracy does not necessarily indicate better chart reading.

  • Structured annotations and targeted controls.

The reference links extraction targets to evidence and rules, while controls on identity writing, output format, and field sets help clarify the sources of improvement.

Critical weaknesses
  • Unclear presentation of the task and experimental design.

The relationships between chart content, record identities, required fields, and the determination-basis ledger are not explained clearly enough. Important methodological details are distributed across the main text and appendices, making it difficult to determine exactly what information each experimental condition provides and which capabilities it evaluates. A worked example tracing a chart through annotation, model input, extraction, and scoring would substantially improve clarity.

  • Limited insight into practical extraction workflows.

The application is potentially useful, but the paper does not clearly connect its controlled experiments to a realistic extraction workflow. The finding that supplying correct record identities or required fields improves accuracy is largely expected. Its practical significance depends on whether this information would normally be available: a predefined field schema may be a realistic input, whereas correct record identities may constitute additional information unavailable before extraction. The paper should distinguish these cases and explain how its findings inform system design when record identities must themselves be inferred or supplied imperfectly. As presented, the results offer limited guidance beyond demonstrating the benefit of additional task information.

  • Unclear relationship between strict accuracy and extraction quality.

The strict accuracy metric penalizes additional fields even when their contents are supported by the source chart but absent from the reference annotations. Such penalties can be appropriate for extraction under a predefined schema, but their interpretation depends on whether the extra fields violate an explicit output specification or instead expose incomplete reference coverage. These cases should be distinguished.

Consequently, improvements under conditions that specify the required fields may primarily reflect better schema compliance rather than more accurate interpretation of chart content. Although Section 5.2 acknowledges this issue, the main conclusions should more clearly separate record matching, field selection, and value correctness. A complementary evaluation that distinguishes unsupported outputs from chart-supported but out-of-scope information would strengthen the interpretation of the results.

Questions for authors

See the Weaknesses for more details.

Reviewer_93cCOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies structured information extraction from FAA instrument approach charts, including fix roles, ordered segments, vertical constraints, and published minima. It constructs a reference linking required records and attributes to chart evidence and rules. Supplying instance lists increases strict accuracy from 40.72% to 71.43% for Luna, 5.76% to 29.15% for Qwen, and 49.18% to 73.70% for K3. Most gains come from previously unmatched targets becoming fully correct.

Critical strengths
  • Separating record recovery, field selection, and value correctness makes aggregate accuracy easier to interpret.
  • A reasonably large evaluation on difficult charts: 862 charts with 14,446 targets. Comparisons across three models use a subset of 171 charts from 106 airports.
  • Requiring models to write the supplied identities is a useful control. The improvement does not depend on the scorer assigning identities through row numbers.
  • The authors clearly acknowledge limitations and report negative results, including the failure of the proposed difficulty scale.
Critical weaknesses
  • Limited methodological novelty. Separating missing records, schema conformity, and incorrect values is a useful but familiar engineering approach.
  • Very specific application, with little evidence that the findings generalize beyond this chart type and evaluation setup.
  • The practical benefit is unclear. The supplied lists already contain much of the structure the model is supposed to extract. The paper does not show how to obtain these lists reliably in practice.
  • Confusing to follow in its current form. Too many conditions, abbreviations, and references between sections. It is often difficult to tell exactly what changes between experiments.
  • The scoring makes some results hard to interpret. Correct information printed on a chart can count as an error if the reference does not annotate that field. Large accuracy gains can therefore reflect agreement with annotation choices.
  • The authors acknowledge many confounds, but this leaves relatively few conclusions that can be stated confidently.
  • False-positive records are not adequately evaluated. Target-based accuracy measures recovery of required records, but the paper does not report comprehensive precision, hallucinated-instance rates, or duplicate rates.
Questions for authors
  • What is the main finding that should generalize to other structured extraction tasks? What evidence supports that generalization?
  • How would performance change with incomplete or incorrect instance lists from an actual extraction pipeline?
  • Why should correct but unannotated chart information make a record incorrect? Do the conclusions hold under a scoring scheme that allows such information?
1 comment

With the discussion period drawing to a close, and after reading other reviewers' reviews, I stand by my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.