- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces LED-BENCH, a benchmark for evaluating multimodal models’ fine-grained understanding and cross-region or cross-view reasoning over complete engineering drawings. It contains 107 drawings and 377 open-ended questions spanning mechanical, architectural, and transportation engineering. Questions are constructed through an expert-led, model-assisted process and categorized by their evidence-integration and reasoning demands. The authors evaluate 11 models over three runs using strict binary scoring determined by a three-LLM judge panel. The best domain-macro accuracy is 52.65%, and a human review of 90 responses achieves 95.6% agreement with automatic scoring. Visual-access interventions further show that providing local crops improves performance, while additional localization or search steps do not consistently produce better results.
Critical strengths
- A practically relevant evaluation setting spanning multiple engineering domains. Engineering-drawing question answering requires combining local annotations, object correspondences, and spatially distributed visual evidence. Evaluating mechanical, architectural, and transportation drawings within a common framework helps reveal differences in model capabilities across engineering representations and provides a useful setting for studying complete-drawing understanding and evidence integration.
- The visual-access interventions offer concrete, actionable insights. Fixed crops improve the four-model mean domain-macro accuracy from 36.38% to 49.34%, approaching the 49.74% achieved by search-then-reread while requiring fewer calls. This moves the paper beyond a model leaderboard and suggests that improving access to local evidence can be an effective and relatively simple direction.
Critical weaknesses
- The evidence supporting reference-answer quality remains insufficiently auditable. The paper states that experts review questions and answers against the original drawings, but does not report the number and qualifications of reviewers, whether each item receives independent verification, or the outcomes of disagreement resolution and error correction. The human review of 90 responses supports agreement between automatic and human scoring, but because the human reviewer judges responses against the reference answers, it does not independently validate those references. For a benchmark paper using strict complete-answer scoring, this gap limits confidence in the benchmark’s reliability and the interpretation of model failure rates.
- Missing reporting of actual scoring coverage makes the comparability of the main results difficult to verify. The paper explains how missing responses and unresolved judge outcomes are handled, but does not report how frequently these cases occur across models, domains, and experimental conditions. Because unresolved judge outcomes are excluded from the denominator, it is necessary to establish whether the reported scores are based on comparable question sets with sufficiently complete coverage. The judge models and versions are also not explicitly identified, limiting reproducibility. These omissions do not establish that the results are biased, but the available evidence is insufficient to assess whether they affect the main comparisons.
Questions for authors
The following are clarification requests concerning the existing data and experiments; their answers could materially affect my assessment.
- How many reviewers with relevant domain expertise checked each reference answer? Was there verification against the original drawing by someone independent of the initial answer-generation process? Please report any existing evidence on disagreements, revisions, or errors identified during verification.
- What is the actual scoring coverage for each model, domain, and experimental condition in the main evaluation and visual-access intervention study? Were any unresolved judge outcomes excluded? If coverage is 100% throughout, explicitly stating this would resolve the concern. Otherwise, do the main conclusions hold when comparisons are restricted to the common set of scorable questions?
- Which models and versions constitute the three-LLM judge panel? Was the same judge configuration used across evaluated models and across the main evaluation and intervention experiments?
4 comments
Response to W2(a), Q2, W2(b) & Q3
W2(a): "Missing reporting of actual scoring coverage makes the comparability of the main results difficult to verify. The paper explains how missing responses and unresolved judge outcomes are handled, but does not report how frequently these cases occur across models, domains, and experimental conditions."
Q2: "What is the actual scoring coverage for each model, domain, and experimental condition in the main evaluation and visual-access intervention study? Were any unresolved judge outcomes excluded? If coverage is 100% throughout, explicitly stating this would resolve the concern. Otherwise, do the main conclusions hold when comparisons are restricted to the common set of scorable questions?"
We have added scoring coverage in Appendix A and condition-level coverage in Appendix F, extending the original scoring and intervention protocols.
In the main evaluation, 12,440 of 12,441 planned model–question–round records have final scores (99.9920%). The single unresolved record is a mechanical Qwen3.6 Flash response with conflicting available judge votes. It remains NA and is excluded from its scored denominator.
The intersection scorable by all eleven models contains 1,130 question–round records per model. On this identical set, the complete model ranking is unchanged, Hard remains below Medium for every model, and the largest absolute change in domain-average accuracy is 0.0496%.
The visual-access study was conducted on the full dataset, covering four models, four visual-access conditions, and three independent runs. Of the 18,096 planned records, 17,837 (98.57%) received valid final scores, while the remaining 259 were marked as NA due to missing or invalid scoring outcomes. Given the high scoring coverage, these unresolved records are unlikely to materially affect the overall conclusions.
Table 1. Scoring coverage across visual-access conditions.
| Condition | Scored / Planned | Coverage | NA |
|---|---|---|---|
| Whole-sheet Direct | 4,457 / 4,524 | 98.52% | 67 |
| Fixed Crops | 4,460 / 4,524 | 98.59% | 64 |
| Locate–Reread | 4,461 / 4,524 | 98.61% | 63 |
| Search–Reread | 4,459 / 4,524 | 98.56% | 65 |
W2(b): "The judge models and versions are also not explicitly identified, limiting reproducibility. These omissions do not establish that the results are biased, but the available evidence is insufficient to assess whether they affect the main comparisons."
Q3: "Which models and versions constitute the three-LLM judge panel? Was the same judge configuration used across evaluated models and across the main evaluation and intervention experiments?"
We have added the judge names in Section 4 and returned identifiers in Appendix A, and clarified the visual-access configuration in Appendix F.
We used three distinct judge models:
- gpt-5.6-terra (
gpt-5.6-terra-2026-07-09) - kimi-k3
- deepseek-v4.1-flash
The same judge panel, binary scoring prompt, and majority-voting procedure were used throughout the evaluation. We have clarified this in Section 4 and Appendix A of our revised paper.
We hope that our responses address your concerns. Please let us know if you have further questions.
Thank you for your response. The demostration of datailed dataset helps me understand the dataset. These datasets are highly valuable for evaluating the engineering drawing comprehension capabilities of large models. The scale of the dataset remains my primary concern; large-scale data testing is crucial for uncovering the capability limitations of LLMs.
Thanks for your feedback.
Thank you very much for your further feedback and recognition of the value of LED-BENCH.
Large-scale evaluation is valuable for comprehensively understanding the capabilities and limitations of large models. At the same time, we believe that the value of a benchmark depends not only on its scale but also on the quality of its questions, the capabilities it evaluates, and the insights it provides.
First, LED-BENCH provides meaningful insights into the limitations of current MLLMs. Our evaluation reveals important challenges in object association and relation reconstruction. In particular, Relation Reconstruction accounts for the largest proportion of classified failures (47.3%). This suggests that current MLLMs still struggle to reconstruct structural, spatial, and functional relationships among engineering objects, which are fundamental to understanding realistic engineering systems. These difficulties may reflect limitations in the models' underlying understanding of physical structures and functional relationships.
Second, LED-BENCH provides a foundation for broader engineering drawing evaluation. Although the current benchmark covers three engineering domains, its expert-guided construction and evaluation methodology can be extended to other drawing-intensive fields, such as electronic circuit schematics and piping and instrumentation diagrams (P&IDs). In this regard, LED-BENCH provides not only a collection of evaluation questions but also a useful reference for developing similar benchmarks in other engineering domains.
We recognize that expanding the dataset would further enrich its evaluation coverage and enable more fine-grained analysis across engineering disciplines, drawing types, and reasoning requirements. We therefore plan to continuously expand LED-BENCH with additional engineering drawings, questions, and domains, while maintaining our emphasis on data quality.
We appreciate your constructive suggestions and remain confident in the value of LED-BENCH for evaluating fine-grained understanding and complex reasoning over real-world engineering drawings.
We hope that our responses has addressed your concerns. Please let us know if you have further questions.
Response to W1 & Q1
We sincerely thank you for taking the time to carefully review our paper and for your constructive comments. Please see our detailed responses below.
We have expanded Appendix E with the expert roles, independent drawing-based verification, and recorded reference-answer corrections.
Each engineering discipline involves three domain experts with over three years of relevant industry experience, with distinct responsibilities:
Evidence Identification: Two experts examine the original drawings and identify the target objects, relevant annotations, engineering relationships, and required answer components.
Question Construction: These expert-defined requirements guide LLM-assisted generation of candidate questions and reference answers.
Independent Verification: A third expert, uninvolved in question generation, checks each QA pair against the original drawing for technical correctness, object–attribute associations, engineering relationships, and answer completeness.
Our verification procedure follows an iterative review-and-revision mechanism. When the independent reviewer identifies an incorrect reference answer, insufficient evidence, an ambiguous target, or missing answer components, the candidate is rejected and returned for revision or regeneration. The revised candidate must be independently reviewed again before acceptance.
We report the outcomes of a clearly identifiable review round. Specifically, the V6–V7 review covered 201 questions across 67 architectural and transportation drawings. Among these questions:
The revisions addressed different issues, including incorrect engineering relationships, ambiguous question targets, unsupported interpretations, redundant information requirements, and incomplete reference answers. Table 1 presents three representative factual corrections identified during drawing-based review.
Table 1. Representative corrections during the construction of LED-BENCH.
In the first case, the original answer incorrectly required movement through the Game Area, whereas the drawing shows a connecting corridor along its southern side.
In the second case, the original answer incorrectly characterized the bridge support configuration, which was corrected after examining the corresponding section.
In the third case, the review corrected the assignment of a network element to a lateral branch rather than the continuous mainline.