acceptodds
Under review as a conference paper at ICLR 2027

LED-BENCH: A Benchmark for Large-format Engineering Drawing Understanding

Abstract

Real-world engineering drawings are often large-format and information-dense. Answering practical engineering questions therefore requires both fine-grained understanding of engineering information and complex reasoning over objects, relations, and evidence distributed across different regions or views. Existing benchmarks cover a range of engineering-drawing understanding tasks involving local regions, individual views, specific elements, or more constrained visual settings. However, systematic evaluation of understanding over realistic, large-format engineering drawings remains limited. To address this gap, we introduce LED-BENCH, a multidisciplinary benchmark designed to evaluate fine-grained understanding and complex reasoning over large-format engineering drawings, spanning the mechanical, architectural, and transportation domains. Extensive evaluation of 11 representative multimodal large language models, including 8 proprietary models and 3 open-source models, reveals substantial remaining difficulty: GPT-5.6 Sol achieves 52.65% average accuracy, performance varies across engineering domains, and all evaluated models perform worse on questions requiring broader evidence integration. These results highlight fine-grained understanding, evidence integration, and complex reasoning over large-format engineering drawings as persistent challenges for current multimodal models. Code and data will be made publicly available.

Primary area:
datasets and benchmarks
Keywords:
Multidisciplinary, Large-format Engineering Drawing

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_PqiFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LED-BENCH, a benchmark for evaluating multimodal models’ fine-grained understanding and cross-region or cross-view reasoning over complete engineering drawings. It contains 107 drawings and 377 open-ended questions spanning mechanical, architectural, and transportation engineering. Questions are constructed through an expert-led, model-assisted process and categorized by their evidence-integration and reasoning demands. The authors evaluate 11 models over three runs using strict binary scoring determined by a three-LLM judge panel. The best domain-macro accuracy is 52.65%, and a human review of 90 responses achieves 95.6% agreement with automatic scoring. Visual-access interventions further show that providing local crops improves performance, while additional localization or search steps do not consistently produce better results.

Critical strengths
  1. A practically relevant evaluation setting spanning multiple engineering domains. Engineering-drawing question answering requires combining local annotations, object correspondences, and spatially distributed visual evidence. Evaluating mechanical, architectural, and transportation drawings within a common framework helps reveal differences in model capabilities across engineering representations and provides a useful setting for studying complete-drawing understanding and evidence integration.
  2. The visual-access interventions offer concrete, actionable insights. Fixed crops improve the four-model mean domain-macro accuracy from 36.38% to 49.34%, approaching the 49.74% achieved by search-then-reread while requiring fewer calls. This moves the paper beyond a model leaderboard and suggests that improving access to local evidence can be an effective and relatively simple direction.
Critical weaknesses
  1. The evidence supporting reference-answer quality remains insufficiently auditable. The paper states that experts review questions and answers against the original drawings, but does not report the number and qualifications of reviewers, whether each item receives independent verification, or the outcomes of disagreement resolution and error correction. The human review of 90 responses supports agreement between automatic and human scoring, but because the human reviewer judges responses against the reference answers, it does not independently validate those references. For a benchmark paper using strict complete-answer scoring, this gap limits confidence in the benchmark’s reliability and the interpretation of model failure rates.
  2. Missing reporting of actual scoring coverage makes the comparability of the main results difficult to verify. The paper explains how missing responses and unresolved judge outcomes are handled, but does not report how frequently these cases occur across models, domains, and experimental conditions. Because unresolved judge outcomes are excluded from the denominator, it is necessary to establish whether the reported scores are based on comparable question sets with sufficiently complete coverage. The judge models and versions are also not explicitly identified, limiting reproducibility. These omissions do not establish that the results are biased, but the available evidence is insufficient to assess whether they affect the main comparisons.
Questions for authors

The following are clarification requests concerning the existing data and experiments; their answers could materially affect my assessment.

  1. How many reviewers with relevant domain expertise checked each reference answer? Was there verification against the original drawing by someone independent of the initial answer-generation process? Please report any existing evidence on disagreements, revisions, or errors identified during verification.
  2. What is the actual scoring coverage for each model, domain, and experimental condition in the main evaluation and visual-access intervention study? Were any unresolved judge outcomes excluded? If coverage is 100% throughout, explicitly stating this would resolve the concern. Otherwise, do the main conclusions hold when comparisons are restricted to the common set of scorable questions?
  3. Which models and versions constitute the three-LLM judge panel? Was the same judge configuration used across evaluated models and across the main evaluation and intervention experiments?
4 comments
AuthorsOpenReview ↗

Response to W1 & Q1

We sincerely thank you for taking the time to carefully review our paper and for your constructive comments. Please see our detailed responses below.


W1: "The evidence supporting reference-answer quality remains insufficiently auditable. The paper states that experts review questions and answers against the original drawings, but does not report the number and qualifications of reviewers, whether each item receives independent verification, or the outcomes of disagreement resolution and error correction. The human review of 90 responses supports agreement between automatic and human scoring, but because the human reviewer judges responses against the reference answers, it does not independently validate those references. For a benchmark paper using strict complete-answer scoring, this gap limits confidence in the benchmark's reliability and the interpretation of model failure rates."

Q1: "How many reviewers with relevant domain expertise checked each reference answer? Was there verification against the original drawing by someone independent of the initial answer-generation process? Please report any existing evidence on disagreements, revisions, or errors identified during verification."

We have expanded Appendix E with the expert roles, independent drawing-based verification, and recorded reference-answer corrections.

Each engineering discipline involves three domain experts with over three years of relevant industry experience, with distinct responsibilities:

  • Evidence Identification: Two experts examine the original drawings and identify the target objects, relevant annotations, engineering relationships, and required answer components.

  • Question Construction: These expert-defined requirements guide LLM-assisted generation of candidate questions and reference answers.

  • Independent Verification: A third expert, uninvolved in question generation, checks each QA pair against the original drawing for technical correctness, object–attribute associations, engineering relationships, and answer completeness.

Our verification procedure follows an iterative review-and-revision mechanism. When the independent reviewer identifies an incorrect reference answer, insufficient evidence, an ambiguous target, or missing answer components, the candidate is rejected and returned for revision or regeneration. The revised candidate must be independently reviewed again before acceptance.

We report the outcomes of a clearly identifiable review round. Specifically, the V6–V7 review covered 201 questions across 67 architectural and transportation drawings. Among these questions:

  • 166 (82.6%) were confirmed without modification.
  • 35 (17.4%) were revised following the review.

The revisions addressed different issues, including incorrect engineering relationships, ambiguous question targets, unsupported interpretations, redundant information requirements, and incomplete reference answers. Table 1 presents three representative factual corrections identified during drawing-based review.

Table 1. Representative corrections during the construction of LED-BENCH.

Question IDOriginal ReferenceCorrected Reference
ARCHITECTURE-011-UNIFIED-HARD-Q1Gymnasium → Game Area → central corridor/vestibule → Meeting RoomGymnasium → corridor along the south side of the Game Area → Meeting Room
TRANSPORT-021-UNIFIED-HARD-Q1Bridge support described as a singular pierFour vertical pile/column supports beneath the bridge deck
TRANSPORT-042-UNIFIED-HARD-Q3W19 identified as a south-side branchW17-1 is the lateral branch; W19 belongs to the continuous W-series mainline

In the first case, the original answer incorrectly required movement through the Game Area, whereas the drawing shows a connecting corridor along its southern side.

In the second case, the original answer incorrectly characterized the bridge support configuration, which was corrected after examining the corresponding section.

In the third case, the review corrected the assignment of a network element to a lateral branch rather than the continuous mainline.

AuthorsOpenReview ↗

Response to W2(a), Q2, W2(b) & Q3

W2(a): "Missing reporting of actual scoring coverage makes the comparability of the main results difficult to verify. The paper explains how missing responses and unresolved judge outcomes are handled, but does not report how frequently these cases occur across models, domains, and experimental conditions."

Q2: "What is the actual scoring coverage for each model, domain, and experimental condition in the main evaluation and visual-access intervention study? Were any unresolved judge outcomes excluded? If coverage is 100% throughout, explicitly stating this would resolve the concern. Otherwise, do the main conclusions hold when comparisons are restricted to the common set of scorable questions?"

We have added scoring coverage in Appendix A and condition-level coverage in Appendix F, extending the original scoring and intervention protocols.

In the main evaluation, 12,440 of 12,441 planned model–question–round records have final scores (99.9920%). The single unresolved record is a mechanical Qwen3.6 Flash response with conflicting available judge votes. It remains NA and is excluded from its scored denominator.

The intersection scorable by all eleven models contains 1,130 question–round records per model. On this identical set, the complete model ranking is unchanged, Hard remains below Medium for every model, and the largest absolute change in domain-average accuracy is 0.0496%.

The visual-access study was conducted on the full dataset, covering four models, four visual-access conditions, and three independent runs. Of the 18,096 planned records, 17,837 (98.57%) received valid final scores, while the remaining 259 were marked as NA due to missing or invalid scoring outcomes. Given the high scoring coverage, these unresolved records are unlikely to materially affect the overall conclusions.

Table 1. Scoring coverage across visual-access conditions.

ConditionScored / PlannedCoverageNA
Whole-sheet Direct4,457 / 4,52498.52%67
Fixed Crops4,460 / 4,52498.59%64
Locate–Reread4,461 / 4,52498.61%63
Search–Reread4,459 / 4,52498.56%65

W2(b): "The judge models and versions are also not explicitly identified, limiting reproducibility. These omissions do not establish that the results are biased, but the available evidence is insufficient to assess whether they affect the main comparisons."

Q3: "Which models and versions constitute the three-LLM judge panel? Was the same judge configuration used across evaluated models and across the main evaluation and intervention experiments?"

We have added the judge names in Section 4 and returned identifiers in Appendix A, and clarified the visual-access configuration in Appendix F.

We used three distinct judge models:

  • gpt-5.6-terra (gpt-5.6-terra-2026-07-09)
  • kimi-k3
  • deepseek-v4.1-flash

The same judge panel, binary scoring prompt, and majority-voting procedure were used throughout the evaluation. We have clarified this in Section 4 and Appendix A of our revised paper.


We hope that our responses address your concerns. Please let us know if you have further questions.

Thank you for your response. The demostration of datailed dataset helps me understand the dataset. These datasets are highly valuable for evaluating the engineering drawing comprehension capabilities of large models. The scale of the dataset remains my primary concern; large-scale data testing is crucial for uncovering the capability limitations of LLMs.

AuthorsOpenReview ↗

Thanks for your feedback.

Thank you very much for your further feedback and recognition of the value of LED-BENCH.

Large-scale evaluation is valuable for comprehensively understanding the capabilities and limitations of large models. At the same time, we believe that the value of a benchmark depends not only on its scale but also on the quality of its questions, the capabilities it evaluates, and the insights it provides.

First, LED-BENCH provides meaningful insights into the limitations of current MLLMs. Our evaluation reveals important challenges in object association and relation reconstruction. In particular, Relation Reconstruction accounts for the largest proportion of classified failures (47.3%). This suggests that current MLLMs still struggle to reconstruct structural, spatial, and functional relationships among engineering objects, which are fundamental to understanding realistic engineering systems. These difficulties may reflect limitations in the models' underlying understanding of physical structures and functional relationships.

Second, LED-BENCH provides a foundation for broader engineering drawing evaluation. Although the current benchmark covers three engineering domains, its expert-guided construction and evaluation methodology can be extended to other drawing-intensive fields, such as electronic circuit schematics and piping and instrumentation diagrams (P&IDs). In this regard, LED-BENCH provides not only a collection of evaluation questions but also a useful reference for developing similar benchmarks in other engineering domains.

We recognize that expanding the dataset would further enrich its evaluation coverage and enable more fine-grained analysis across engineering disciplines, drawing types, and reasoning requirements. We therefore plan to continuously expand LED-BENCH with additional engineering drawings, questions, and domains, while maintaining our emphasis on data quality.

We appreciate your constructive suggestions and remain confident in the value of LED-BENCH for evaluating fine-grained understanding and complex reasoning over real-world engineering drawings.

We hope that our responses has addressed your concerns. Please let us know if you have further questions.

Reviewer_UTNuOpenReview ↗
Rating
2(+1)
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces LED-Bench, a benchmark that focuses on engineering drawing spans mechanical, architecture, and transportation three domains. The benchmark contains overall 107 images and 377 question, labeled by domain experts, to evaluate fine-grained visual understanding and complex cross-region reasoning over large-format drawings. In experiments, it evaluates 3 GPT versions, 4 Qwen versions, and Kimi and Gemini. The results suggest that current multimodal models struggle significantly with complete engineering sheets, facing major bottlenecks in visual evidence integration and detail retrieval.

Critical strengths
  • The author claims that the data is well designed, selected, and evaluated by domain experts, spanning medium and hard two difficulty levels.
  • The benchmarking results can align with insights.
  • Clearly isolates model failure modes across four distinct stages and provides practical visual-access interventions (e.g., crop-and-reread strategies) to guide future multimodal architectures.
Critical weaknesses
  • The collected data size is very limited, which contains only 107 images and 377 questions, and finally, making only about 100 questions per domain. The limited data size weakens the soundness.
  • The work only evaluated four types of models, Kimi, Qwen, Gemini, and GPT. More models and expertised model should be included.
  • The evaluation lacks domain-related design, making the final evaluation easily biased.
  • The issues and gaps of previous works are not clearly stated; the motivation is not convincing.
Questions for authors

Try to address weaknesses.

4 comments
AuthorsOpenReview ↗

Response to W1 and W2

We sincerely thank you for your time devoted to reviewing this paper and your constructive comments. Please see our detailed responses to your comments and suggestions below.


W1: "The collected data size is very limited, which contains only 107 images and 377 questions, and finally, making only about 100 questions per domain. The limited data size weakens the soundness."

We have expanded the construction details in Appendix E with expert roles and review outcomes

To address this concern, we would like to further clarify the scale and representativeness of our benchmark.

LED-BENCH provides a substantially larger evaluation set than closely related expert-curated drawing QA benchmarks. Our benchmark contains 107 complete, large-format engineering drawings and 377 QA pairs across three engineering disciplines, exceeding both benchmarks in the number of drawings and questions.

  • DrawingVQA: 33 drawings and 92 QA pairs.
  • AECV-Bench: 21 drawings and 192 QA pairs.

More importantly, LED-BENCH consists of carefully designed, high-quality questions specifically targeting real-world engineering drawing understanding. Each question is grounded in practical engineering requirements, emphasizing complex and meaningful reasoning, and is independently verified against the original drawing by an experienced domain expert.

In contrast, the larger-scale MechVQA is primarily designed for model training, with 10% of its 20,778 QA pairs allocated to testing. Its automatically generated questions are primarily validated for correctness, answerability, and consistency.

We recognize the importance of further improving drawing diversity and expanding question coverage, and consider these important directions for future work.


W2: "The work only evaluated four types of models, Kimi, Qwen, Gemini, and GPT. More models and expertised model should be included."

We have expanded the original eleven-model results to fourteen models in Section 4 and extended the existing repeated-run tables in Appendix C.

We added MiniMax-M3, Gemma 4-31B-it, and the mechanical drawing specialized model MechVL-4B-RL, expanding the evaluation from eleven to fourteen models. Their results are presented in the following tables.

Table 1. Model performance across engineering domains.

ModelMechanicalArchitectureTransportation
Open-source MLLMs
MiniMax-M344.64%24.44%37.24%
Gemma 4-31B-it17.62%8.48%9.42%
MechVL-4B-RL8.62%5.78%4.69%

Table 2. Run-to-run stability.

ModelR1R2R3Mean ± SD
Open-source MLLMs
MiniMax-M336.60%40.32%37.40%38.11 ± 1.96%
Gemma 4-31B-it14.06%12.23%12.80%13.03 ± 0.93%
MechVL-4B-RL6.90%6.63%6.63%6.72 ± 0.15%

MechVL-4B-RL is the only model specifically adapted to mechanical drawing understanding. Despite this specialization, the 4B-parameter model performs poorly on the complex questions in LED-BENCH.

AuthorsOpenReview ↗

Response to W3 and W4

W3: "The evaluation lacks domain-related design, making the final evaluation easily biased."

We have expanded Appendix E with domain-specific question-construction requirements.

We thank the reviewer for this valuable comment. LED-BENCH incorporates domain-specific engineering requirements into its question construction to reflect the distinct characteristics of different engineering disciplines.

Specifically, for each engineering discipline, we design tailored question formulations, reasoning requirements, and reference answer components based on its specific engineering tasks through our expert-guided, model-assisted construction process.

Although these disciplines differ in drawing conventions and engineering semantics, they share fundamental reasoning requirements, as illustrated in Figure 3(b). These shared capabilities form the core focus of LED-BENCH's cross-disciplinary evaluation.


W4: "The issues and gaps of previous works are not clearly stated; the motivation is not convincing."

We have expanded the original related-work comparison in Appendix E.

Existing engineering drawing benchmarks have made important contributions, but several capabilities essential to practical engineering interpretation remain insufficiently evaluated, particularly the association of distributed annotations with engineering objects, reconstruction of structural and functional relationships, and integration of multiple technical constraints across complete drawings.

We clarify how LED-BENCH addresses these complementary evaluation needs:

  • MechVQA: Its scope is restricted to the mechanical engineering domain and does not address complete, large-format drawings. LED-BENCH complements this scope by emphasizing cross-region and cross-view evidence integration over complete, large-format drawings across three engineering disciplines.

  • DrawingVQA: Introduces authentic construction drawings and expert-designed reasoning questions. However, 90 of its 92 questions are multiple-choice. LED-BENCH instead uses 377 open-ended questions, requiring models to independently construct complete, drawing-grounded answers.

  • AECV-Bench: Evaluates OCR, counting, spatial reasoning, and comparison. LED-BENCH places additional emphasis on functional engineering relationships and constraints.

Taken together, these comparisons highlight the need for a systematic evaluation of fine-grained understanding and complex reasoning over realistic, large-format engineering drawings, particularly in associating engineering information with objects and integrating relations and evidence across different regions or views. LED-BENCH is specifically designed to address this gap.


We hope that our responses address your concerns. Please let us know if you have further questions.

I appreciate the response and revisions. The added baselines are helpful for understanding the datasets. The revised motivation and insights also improve the presentation. However, I still think the data size is too limited to generalize to broader domains, especially for applications in current large-model training, where the authors claimed applicability. It would be great to show its capability for post-training. I have increased my score, but I still lean negative due to the limited application scope and the limited insights obtained from the benchmarking results.

AuthorsOpenReview ↗

Thanks for your feedback.

We appreciate your further feedback and recognition of our revisions. We would like to clarify two points regarding the scope and insights of LED-BENCH.

First, LED-BENCH is designed as an evaluation benchmark. Its primary goal is to evaluate fine-grained understanding and complex reasoning over real-world engineering drawings. Although the current benchmark covers three engineering domains, its expert-guided construction and evaluation methodology can be extended to other drawing-intensive fields, such as electronic circuit schematics and piping and instrumentation diagrams (P&IDs), where understanding object associations and structural relationships is equally important. In this regard, LED-BENCH provides a useful reference for developing similar evaluation benchmarks in other engineering domains.

Second, our evaluation provides insights into the limitations of current MLLMs beyond overall accuracy. In our failure-stage analysis, Relation Reconstruction accounts for the largest proportion of classified failures (47.3%). This suggests that although MLLMs can recognize engineering objects and annotations, they still struggle to reconstruct the structural, spatial, and functional relationships among them. Such relationships are fundamental to understanding realistic engineering systems, and these failures may reflect limitations in the models' underlying engineering and physical understanding.

These findings also point to directions for future research. We plan to explore post-training methods that strengthen engineering relationship understanding, potentially incorporating CAD and engineering simulation tools to improve the consistency and reliability of model understanding and reasoning.

Overall, LED-BENCH provides a valuable diagnostic evaluation setting for identifying limitations in current MLLMs and guiding future research toward more reliable engineering drawing understanding.

We hope these clarifications address your concerns. Please let us know if you have further questions.

Reviewer_RTUGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides a visual understanding benchmark for engineering drawings, covering mechanical design, architecture, and transportation.

Critical strengths
  • Engineering drawing benchmark is highly demanding for engineering research community.
  • Writing is clear.
Critical weaknesses

Contributions need more detailed support

  1. This paper highlights its advantage in fine-granularity and complex reasoning. However, it needs more quantitative statistics to support. For example, this paper has elaborated what "fine-grained" means (dimensions, materials, symbols, etc). But the paper does not compare their popularity in its benchmark against prior benchmarks. So readers do not know if this paper provide more or less fine-grained benchmark questions.

  2. Similarly, while it is OK to skip a definition of "complex reasoning", this paper still needs to provide some criteria (e.g. number of cross-diagram reasoning) to convince readers that it really highlights more complex reasoning than prior work.

  3. Finally, this paper has not present a complete example of its benchmark item, which makes it difficult for readers to appreciate the benchmark quality.

Questions for authors
  • Provide statistics comparison to prior benchmarks to showcase your "fine-grained" and "complex reasoning" highlights, as I indicated in the weakness section.
  • Provide complete benchmark examples, ranging from images (if possible) to the related QAs.
6 comments
AuthorsOpenReview ↗

Response to W(1) and Q1(a)

We sincerely thank you for your time devoted to reviewing this paper and your constructive comments. Please see our detailed responses to your comments and suggestions below.

W1: "This paper highlights its advantage in fine-granularity and complex reasoning. However, it needs more quantitative statistics to support. For example, this paper has elaborated what "fine-grained" means (dimensions, materials, symbols, etc). But the paper does not compare their popularity in its benchmark against prior benchmarks. So readers do not know if this paper provides more or less fine-grained benchmark questions."

Q1(a): "Provide statistics comparison to prior benchmarks to showcase your "fine-grained" and "complex reasoning" highlights, as I indicated in the weakness section."

We have expanded the original benchmark comparison with statistics and representative QA examples in Appendix E.

The "fine-grained understanding" emphasized by LED-BENCH goes beyond the simple recognition of dimensions, annotations, or structural components. It requires the model to accurately link multiple engineering attributes to their corresponding objects and to grasp the practical engineering implications conveyed by these associations.

While existing benchmarks do incorporate recognition and reasoning tasks, they often place limited focus on precise object association and the interpretation of engineering relationships. LED-BENCH centers its problem design on these very aspects: a model must not only correctly extract engineering data but also determine which objects that data pertains to, and how these elements collectively form geometric constraints, structural relationships, or functional requirements.

To quantify this distinction, we examine two requirements corresponding to Object Association and Relation Reconstruction in LED-BENCH:

  • Object Association: Establishing multiple necessary associations between engineering objects and their dimensions, specifications, or annotations.

  • Relation Reconstruction: Reconstructing and interpreting geometric, structural, or topological relationships among engineering objects.

Table 1. Proportion of Questions Requiring These Two Capabilities in Existing Benchmarks.

BenchmarkObject AssociationRelation Reconstruction
MechVQA subset (119 questions)24/119 (20.2%)11/119 (9.2%)
DrawingVQA16/92 (17.4%)12/92 (13.0%)
AECV-Bench46/192 (24.0%)22/192 (11.5%)

To illustrate this distinction more clearly, we compare representative QA examples from LED-BENCH and existing engineering drawing benchmarks in Table 2.

Table 2. Representative QA examples.

BenchmarkQuestionAnswer
MechVQAWhat is the diameter dimension of the inner hole in the main view?φ35
AECV-BenchWhich room has an openable door to the TERRASSE?The CUISINE
DrawingVQAAre there any welding done around the beam?Yes.
LED-BENCHHow are the six circumferentially arranged threaded holes in the Main View positioned and geometrically controlled? Specify their thread depth, drilled-hole depth, angular spacing, and positional tolerance.The six M8 threaded holes have a thread depth of 10 mm, a drilled-hole depth of 15 mm, an angular spacing of 60°, and a positional tolerance of ⌀0.5 mm relative to datum D

Please see Rebuttal Figures PDF, p. 1 in the supplementary materials, for the drawing corresponding to this our example.

In LED-BENCH, this question addresses the practical need to determine a hole group’s machining and inspection requirements, requiring fine-grained interpretation of thread depth, drilled-hole depth, angular spacing, and positional tolerance, together with their association with the correct holes and reference datum. These details collectively dictate the methods for manufacturing and inspecting the hole patterns what constitutes a genuine engineering problem.

AuthorsOpenReview ↗

Response to W2, W3 and Q2

W2: "Similarly, while it is OK to skip a definition of 'complex reasoning', this paper still needs to provide some criteria (e.g. number of cross-diagram reasoning) to convince readers that it really highlights more complex reasoning than prior work."

We have added quantitative evidence and reasoning measurements in Appendix E.

To characterize the complexity of engineering reasoning in LED-BENCH, we introduce four quantitative measurements:

  • Evidence Units: The number of distinct pieces of information required to answer the question, such as dimensions, component labels, geometric properties, or engineering constraints.

  • Association Operations: The number of object–attribute bindings or object–object relationships that must be established to correctly connect and interpret the evidence, including correspondences across views.

  • Reasoning Steps: The maximum number of sequentially dependent reasoning steps required to derive the answer, where the result of one step is needed for the next.

  • Spatial Regions: The number of distinct regions of the drawing that contain necessary evidence.

Since there is no widely accepted objective standard for engineering drawing difficulty, the Medium/Hard labels were assigned by engineering experts based on a holistic assessment of each question's evidence and reasoning requirements. The four quantitative dimensions are not independent, and no single dimension can fully capture overall difficulty.

Our subsequent analysis reveals clear quantitative differences: Hard questions involve 52.3% more evidence units, 82.5% more association operations, and 34.8% greater reasoning depth on average than Medium questions. Spatial Regions shows a similar trend, averaging 3.79 for Hard questions compared with 2.29 for Medium questions. Table 3 compares their mean values between Medium and Hard questions.

Table 3. Quantitative requirements across Medium and Hard questions.

MeasurementMedium meanHard mean
Evidence Units3.966.03
Association Operations1.893.45
Reasoning Steps1.782.40
Spatial Regions2.293.79

In Table 2, we compare LED-BENCH with questions labeled as Level 3 (Expert-level) in DrawingVQA, as well as representative questions from other benchmarks, to further highlight the complex engineering reasoning required by our questions.


W3: "Finally, this paper has not present a complete example of its benchmark item, which makes it difficult for readers to appreciate the benchmark quality."

Q2: "Provide complete benchmark examples, ranging from images (if possible) to the related QAs."

We have added complete drawing–question–answer examples for all three disciplines in Appendix E.

Representative examples include:

Mechanical

See Rebuttal Figures PDF, p. 2, for the drawing corresponding to this example.

Question: Based on the assembly drawing, sort out the gear transmission chain of the power take unit: After the external gears enter the power take unit, which gears inside the unit form a transmission relationship with them? Please provide the part numbers and tooth counts of the relevant gears, and explain to which shaft the power is ultimately transmitted.

Answer: External m=4, z=38 gear → No. 7 gear → No. 14 gear → No. 22 spline shaft.

Architecture

See Rebuttal Figures PDF, p. 3, for the drawing corresponding to this example.

Question: How are the new air-distribution branches and the two restroom exhaust paths tied back into the existing HVAC systems?

Answer: New supply/return branches connect to the existing air handler; Men's Toilet 104 → EF-1 → existing exhaust duct; Women's Toilet 103 → EF-2 → existing exhaust duct.

Transportation

See Rebuttal Figures PDF, p. 4, for the drawing corresponding to this example.

Question: For the drainage route from roadway gutter to the receiving basin/pond, what path is followed and what net elevation drop is shown from detail point B to point F?

Answer: Gutter → flume → basin/pond; point B is 965.58 ft and point F is 965.22 ft, for a 0.36-ft drop.


We hope that our responses address your concerns. Please let us know if you have further questions.

Thanks for your response. Could you please add the statistics of LEDBench to Table 1 "Proportion of Questions Requiring These Two Capabilities in Existing Benchmarks" ?

Reviewer_kUn2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LED-BENCH, a benchmark for evaluating multimodal large language models on complete, large-format engineering drawings. It contains 107 drawing sheets and 377 questions across mechanical, architectural, and transportation engineering. The benchmark focuses on fine-grained information understanding and reasoning over evidence distributed across different regions or views. The authors evaluate 11 MLLMs using a three-run protocol and three-judge binary scoring, with the strongest model achieving 52.65% domain-macro accuracy.

Critical strengths
  • Understanding information distributed across multiple views, sections, dimensions, and annotations within a complete engineering drawing is an important capability for future AI-assisted engineering design workflows. Evaluating complete sheets rather than exclusively localized visual regions provides a useful and more realistic setting for studying these capabilities.

  • The comparison of visual-access strategies is an interesting observation and should be useful for follow-up work. Simple fixed overlapping crops perform almost as well as the multi-step search-then-reread strategy (49.34% vs. 49.74%). Locate-then-reread actually hurts on mechanical drawings. This suggests that much of the current difficulty lies in getting the right pixels to the model, and it gives future methods a cheap baseline to beat.

Critical weaknesses
  • For a benchmark built around fine-grained questions, a two-level Medium/Hard label carries limited information. By the paper’s own definition, Medium can already include simple cross-region or cross-view correspondence. Hard mixes several distinct factors: evidence dispersion, topological reconstruction, multi-step calculation, and multiple constraints. As a result, the Medium–Hard gap does not tell us which of these factors makes questions hard, and it does not directly support the cross-region/cross-view contribution the paper emphasizes. Figure 3(b) proposes a capability hierarchy, but the questions are not annotated along it. Finer per-question annotations would make the contribution much clearer and the dataset more useful for future analysis. Examples include the capability type, the number of evidence regions or views involved, and whether the evidence is local or distributed.

  • The automatic scoring protocol is relatively well documented, including the judge prompt, majority-voting procedure, and a blinded human validation study. However, the exact models and versions used as automatic judges are not explicitly identified. Since the reported benchmark results depend on LLM-based semantic grading, this information is important for reproducibility.

  • The failure-stage analysis is potentially the most informative part of the paper, especially the finding that nearly half of the errors (47.3%) come from relation reconstruction. However, the paper does not report several things: how many errors were coded, how they were sampled and from which models, how a response with several errors was assigned to a single stage, and whether different annotators agreed. Appendix D only gives the stage definitions. Without these details, it is hard to judge how reliable the reported distribution is.

Questions for authors
  • Sec. 3.3 combines several capability and evidence factors in the Medium/Hard labels. Could you provide question counts and model accuracy by capability type and evidence distribution? In particular, how many questions require evidence from multiple regions or views?

  • Which exact models and versions served as the three automatic judges in Sec. 4.1 and Appendix A.2? Were these different models or separate calls to the same model? Was the same judge configuration used throughout the evaluation?

  • For the failure-stage distribution in Figure 4(a), how many errors were coded, and how were they sampled across models and domains? How were responses with multiple errors assigned to one stage? Please also clarify the number of annotators and whether inter-annotator agreement was measured.

3 comments
AuthorsOpenReview ↗

Response to W1(a)

All authors are grateful for your time devoted to reviewing this paper and for your positive feedbacks about our work. Please see our detailed responses to your comments and suggestions below.

W1(a): "Medium can already include simple cross-region or cross-view correspondence. Hard mixes several distinct factors: evidence dispersion, topological reconstruction, multi-step calculation, and multiple constraints. As a result, the Medium–Hard gap does not tell us which of these factors makes questions hard, and it does not directly support the cross-region/cross-view contribution the paper emphasizes. "

We have added the quantitative measurements and the Medium/Hard comparison in Appendix E, complementing the original qualitative definitions in Section 3.

The Medium/Hard classification characterizes overall engineering task difficulty, incorporating the combined demands of evidence integration, structural relationships, and reasoning dependencies. As there is no widely accepted objective standard for engineering drawing difficulty, the Medium/Hard labels were assigned by engineering experts based on a holistic assessment of each question's evidence and reasoning requirements.

To quantify these requirements, we characterize each question using four quantitative measurements:

  • Evidence Units: The number of distinct pieces of information required to answer the question, such as dimensions, component labels, geometric properties.

  • Association Operations: The number of object–attribute bindings or object–object relationships that must be established to correctly connect and interpret the evidence.

  • Reasoning Steps: The maximum number of sequentially dependent reasoning steps required to derive the answer, where the result of one step is needed for the next.

  • Spatial Regions: The number of distinct regions of the drawing that contain necessary evidence.

These four quantitative dimensions are not independent, and no single dimension can fully capture overall difficulty. Our subsequent analysis reveals clear quantitative differences: Hard questions involve 52.3% more evidence units, 82.5% more association operations, and 34.8% more reasoning steps on average than Medium questions. Table 1 compares the mean values of these four measurements between Medium and Hard questions.

Table 1. Quantitative requirements across Medium and Hard questions.

MeasurementMediumHard
Evidence Units3.966.03
Association Operations1.893.45
Reasoning Steps1.782.40
Spatial Regions2.293.79
AuthorsOpenReview ↗

Response to W1(b) and Q1

W1(b): "Finer per-question annotations would make the contribution much clearer and the dataset more useful for future analysis. Examples include the capability type, the number of evidence regions or views involved, and whether the evidence is local or distributed."

Q1: "Could you provide question counts and model accuracy by capability type and evidence distribution? In particular, how many questions require evidence from multiple regions or views?"

We have added capability types, question counts, and model accuracy and local versus distributed evidence statistics in Appendix E.

To address your concerns, we provide finer-grained annotations and quantitative analyses, examining both the capability types and the evidence distribution.

1. Capability Types

To further characterize the engineering capabilities evaluated in LED-BENCH, we categorize all questions into three capability types:

  • Engineering Requirements: Engineering specifications, attributes, and technical requirements, and how they apply to specific components.

  • Spatial Constraints: Dimensions, positions, orientations, and geometric relationships among engineering elements.

  • Functional Configuration: How engineering components are connected, assembled, and organized to perform specific functions.

Table 2. Model accuracy by capability type (%).

ModelEngineering Requirements (126)Spatial Constraints (136)Functional Configuration (115)
Closed-source MLLMs
GPT-5.6 Sol55.8255.1550.43
GPT-5.6 Luna46.8349.7546.09
Qwen3.8-Max46.5647.7945.22
Gemini 3.5 Flash55.8246.0829.57
GPT-5.535.9837.9933.62
Qwen3.7-Flash30.1632.1120.00
Qwen3.8-Flash33.6025.4919.42
Qwen3.6-Flash18.7815.2010.43
Open-source MLLMs
Kimi-K2.532.5427.7020.29
Qwen3.8-27B21.9627.2117.97
Qwen3.6-27B29.6328.4311.88

2. Evidence Distribution

We use Spatial Regions, one of the four quantitative measurements introduced above, to characterize how the evidence required to answer each question is spatially distributed across a drawing.

Spatial Regions: The number of distinct regions of the drawing that contain necessary evidence.

Table 3. Model accuracy by number of spatial regions (%).

Model1 Region (73)2 Regions (113)≥3 Regions (191)
GPT-5.6 Sol62.9252.7648.53
GPT-5.6 Luna55.7948.0441.78
Qwen3.8-Max52.0953.3141.70
Gemini 3.5 Flash61.6541.9234.69
GPT-5.538.0041.0625.17
Qwen3.7-Flash26.8330.5721.19
Qwen3.8-Flash26.8532.1619.58
Qwen3.6-Flash16.8515.979.43
Kimi-K2.527.0028.7920.46
Qwen3.8-27B21.4328.0718.91
Qwen3.6-27B24.4723.6716.20

As shown in Table 3, 80.6% of the questions require evidence integration across multiple spatial regions. These results highlight the challenges of associating and integrating spatially distributed engineering evidence across complete, large-format drawings.

AuthorsOpenReview ↗

Response to W2 and Q2, W3(a) and Q3(a), W3(b), W3(c) and Q3(b)

W2: "The exact models and versions used as automatic judges are not explicitly identified. Since the reported benchmark results depend on LLM-based semantic grading, this information is important for reproducibility."

Q2: "Which exact models and versions served as the three automatic judges in Sec. 4.1 and Appendix A.2? Were these different models or separate calls to the same model? Was the same judge configuration used throughout the evaluation?"

We have added the judge names to Section 4 and their returned identifiers to the existing scoring protocol in Appendix A.

We used three distinct judge models: gpt-5.6-terra (gpt-5.6-terra-2026-07-09), kimi-k3, and deepseek-v4.1-flash. The same judge panel, binary scoring prompt, and majority-voting procedure were used throughout the evaluation.


W3(a): "How many errors were coded, how they were sampled and from which models."

Q3(a): "How many errors were coded, and how were they sampled across models and domains?"

We have specified the classification denominator in Section 4 and added the error population and model/domain breakdown in Appendix D.

We evaluated nine MLLMs (GPT-5.6 Sol, GPT-5.6 Luna, GPT-5.5, Gemini 3.5 Flash, Qwen3.8-Max, Qwen3.8-Flash, Qwen3.7-Flash, Qwen3.6-Flash, and Qwen3.8-27B) across three independent runs, yielding 9 × 377 × 3 = 10,179 responses.

Of the 6,581 errors, 5,496 were assigned to failure stage analysis. The remaining 1,085 errors were excluded because their responses did not provide enough information to identify a specific failure stage, such as substandard answers.


W3(b): "How a response with several errors was assigned to a single stage."

Q3(b): "How were responses with multiple errors assigned to one stage?"

We have added the label procedure and the evidence required for each assignment in Appendix D, extending the stage definitions already provided in Appendix D.

We developed an automated hierarchical diagnostic procedure based on the four capability levels in Figure 3(b), with each level mapped to a corresponding failure stage:

  • Perception + Recognition → Local Evidence Understanding.
  • Cross-view / Cross-region Association → Object Association.
  • Topology Understanding → Relation Reconstruction.
  • Integrated Engineering Reasoning → Integrated Engineering Reasoning.

When multiple errors occur, we assign the label to the lowest-level verifiable error in the hierarchy. For example, if a misread dimension leads to an incorrect calculation, the failure stage is Local Evidence Understanding.


W3(c): "And whether different annotators agreed."

We have added the reported double-annotation sample and agreement statistics in Appendix D.

We conducted a double-annotation study on 150 sampled incorrect responses, comprising 56 mechanical, 47 architectural, and 47 transportation responses. Two annotators independently assigned error labels according to the four-stage taxonomy. They agreed on 131 of the 150 responses (87.33%), yielding a Cohen's κ of 0.829.


We hope that our responses address your concerns. Please let us know if you have further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their time and constructive comments. Relative to the original submission, we have made the following revisions. Changes in the revised paper are marked in blue.

On the one hand, we have revised the main paper to improve its clarity and evaluation coverage:

  • In Section 4, we have expanded the original eleven-model evaluation to fourteen models by adding MiniMax-M3, Gemma 4-31B-it, and the specialized MechVL-4B-RL model. (UTNu)

  • In Section 4, we have added the judge names and clarified the scoring protocol. (kUn2, PqiF)

  • In Section 4, we have clarified the failure-stage analysis and the reported error distribution. (kUn2)

On the other hand, we have expanded the appendices to provide additional details and analyses:

  • In Appendix A, we have added the API-returned judge identifiers and further clarified the scoring protocol. (kUn2, PqiF)

  • In Appendix C, we have extended the repeated-run tables to include the three additional models. (UTNu)

  • In Appendix D, we have expanded the failure-stage definitions and provided details on the coding population, evidence used for label assignment, handling of multiple errors, and inter-annotator agreement. (kUn2)

  • In Appendix E, we have expanded the description of benchmark construction to include expert roles, independent verification, domain-specific requirements, and documented reference-answer corrections. (UTNu, PqiF)

  • In Appendix E, we have added benchmark comparisons, three engineering capability types with question counts and model accuracy, quantitative measurements of evidence and reasoning requirements, and statistics on local versus distributed evidence. (kUn2, RTUG, UTNu)

  • In Appendix E, we have provided complete drawings for the hole-group example and representative QA examples from the mechanical, architectural, and transportation domains. (RTUG)

  • In Appendix F, we have updated the visual-access results. (PqiF)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.