- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper presents a specification-grounded reference and diagnostic methodology for structured extraction from FAA instrument approach charts. It evaluates Luna on 862 charts and compares three deployments on a common 171-chart subset, separating identity matching, field-set agreement, and value correctness. The target-level gain accounting and fixed-denominator field diagnostics provide useful insights beyond aggregate accuracy. Overall, I lean toward rejection. Although the diagnostic contribution is valuable, inconsistent control rates and an insufficiently explained scored-field policy leave the central interpretations unresolved, collection-time confounding further limits intervention attribution.
Critical strengths
- The task contract clearly distinguishes identity matching, supplied-entry responses, and strict correctness over all targets.
- The paper distinguishes retrospective analyses from registered hypotheses and reports the unsupported difficulty-scale hypothesis directly.
Critical weaknesses
-
Inconsistency evaluation data. My main concern is the inconsistency between Luna’s control matching rates in Secs. 6.1 and 6.3 and Tables D3, D8, and D9. Free-generation matching is reported as 40.50%, and adding reading rules changes it by +2.33 points, implying 42.83%. The free-plus-reading-rules control used against L4-self instead has 45.71% matching. Both comparisons specify 171 charts and 2,741 targets, so rounding cannot explain the difference. It remains unclear whether these results use different saved controls, condition definitions, or scoring versions. This makes me concerned about the accuracy of the evaluation data involved in this manuscript.
-
Unclear scored-field policy. In Sec. 5.2 and Appendix A.2, the course printed beside the AWONE→ROCCK segment is unannotated and is therefore penalized as an extra field, but the paper does not explain why it is excluded from scoring. Because the same reference generates both the scoring targets and the supplied information, this policy directly determines strict failures and field-list gains. Symmetric deletion measures the resulting score difference, but it does not show whether the scored-field set adequately represents the declared scope. Appendix B also leaves 7 of 251 missed-approach-point discrepancies unexplained. This makes me concerned that the reported gains partly reflect annotation choices rather than model behavior.
Questions for authors
Is the dominant failure a model failure or a task-specification failure? The base prompt leaves four identity boundaries unspecified (Appendix A.1). Stating them (C1) raises Luna’s matching from 40.50% to 59.03%, and 26.95% of unmatched targets already have a key-compatible output. How much of the 85.28% “unformed instances” is a convention mismatch rather than a recovery failure, and why does the main condition omit these conventions?
What outcome would have contradicted the gain accounting? Since the instance list supplies identities verbatim, higher matching is expected by construction. What matters is accuracy on newly matched targets: about 60% (873 of ~1,466) for Luna, versus 89% (1,105/1,242) for targets matched in both conditions. Does this gap not suggest these targets are also harder to read?
With the discussion period drawing to a close, and after reading other reviewers' reviews, I stand by my original rating.