acceptodds
Under review as a conference paper at ICLR 2027

OptoFig: Can Vision-Language Models Read Optoelectronics?

Abstract

Vision–language models (VLMs) offer new opportunities for automating scientific research, including the design of optoelectronic devices. Realizing this potential requires understanding the figures through which such devices are designed and evaluated, from circuit schematics and device structures to the related performance, such as spectra, mode profiles and timing diagrams. However, existing benchmarks do not capture this capability. We introduce OptoFig, a benchmark of 1,092 questions over 799 figures of optoelectronic devices, covering both how the devices are built and their corresponding performance characterization. OptoFig is simulation-grounded and rigorously validated: every answer is measured on the figure itself, read from the simulator or drawing-script record, or written by a domain expert, and passes automated checks and multi-stage expert review. Its 39 task types span three capability categories: perception and recognition, quantitative and comparative, and physical and functional reasoning, with difficulty defined by construction parameters. Experimental results across seven state-of-the-art VLMs reveal substantial room for improvement: overall accuracy ranges from 54.2% to 74.5%, no single model leads across capabilities, and average accuracy is 10.2 points lower on hard items than on easy ones. The experiments further reveal two findings. First, models perceive figures with limited precision: they capture coarse visual patterns but miss the exact quantities a question depends on. Second, models lack the reasoning needed to interpret what they perceive: on the physics questions, even when a model reads a figure correctly, it draws the wrong physical conclusion about one time in three (67.3% correct on average). VLMs' limitations on these device figures thus lie both in the precision of perception and in reasoning from what they see to the underlying physics.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
benchmark, vision-language models, optoelectronic devices, scientific figure understanding, physical reasoning, AI for science

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6TeBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a vision-language model (VLM) evaluation benchmark for optoelectronic device image understanding, aiming to evaluate the ability of existing VLMs to understand device design images, performance characterization images, and related physical information. The dataset contains 799 images and 1,092 question-answer samples, covers 39 task types, and is mainly divided into three categories: visual perception and recognition, quantitative analysis and comparison, and physical and functional reasoning. The authors use simulation data, drawing scripts, and domain expert review to construct standard answers, and divide task difficulty through preset construction parameters, so as to improve the reliability and interpretability of the evaluation.

In terms of experiments, the authors evaluate seven mainstream VLMs, with overall accuracy ranging from 54.2% to 74.5%. Through the analysis of different task types, difficulty levels, and paired questions, the authors point out that existing models have two main limitations: first, they can recognize the overall features of images, but perform insufficiently in precise counting and numerical reading; second, even if they correctly read the image information, they may still fail to further draw correct physical conclusions.

Overall, this paper provides a dedicated benchmark of certain value for evaluating the image understanding ability of VLMs in the optoelectronics field. However, the dataset still has limitations in task distribution and coverage of complex physical reasoning, the effectiveness of the difficulty division has not been fully verified, and the experimental support for conclusions related to physical reasoning is also relatively limited. Therefore, the current experimental results can reflect the shortcomings of models on specific tasks, but are not yet sufficient to fully prove the representativeness of this benchmark for broader optoelectronic scientific research scenarios.

Critical strengths
  1. A domain-targeted evaluation benchmark that can assess different capabilities of VLMs in optoelectronic image understanding. This paper constructs an evaluation benchmark specifically for optoelectronic devices, covering multiple image types such as device structures, optical field distributions, spectra, and circuit diagrams, and divides 39 tasks into three categories: visual perception and recognition, quantitative analysis and comparison, and physical and functional reasoning. Compared with general scientific image understanding benchmarks, this dataset has clearer domain specificity and can provide a reference for analyzing differences in VLM capabilities on optoelectronics-related tasks.

  2. The dataset construction and validation process is relatively standardized, and it provides experimental analysis with certain interpretability. The authors establish standard answers by combining simulation results, drawing scripts, and domain expert review, reducing the risk of errors caused by relying entirely on automatic model-generated annotations. At the same time, they divide task difficulty through preset construction parameters and use paired questions to analyze the difference between visual understanding and subsequent physical reasoning. The related experiments not only report the overall accuracy of the models but also reveal the insufficient precise counting ability and the phenomenon that physical reasoning errors may still occur after correctly reading images. These analyses help to understand the capability limitations of existing VLMs in a more detailed way.

Critical weaknesses
  1. The representativeness of the dataset and the coverage of tasks are limited, especially the lack of sufficient complex physical reasoning tasks. Although OptoFig covers a variety of optoelectronics-related images and 39 task types, the task distribution of the dataset is obviously unbalanced. Among them, electrical-related questions account for about 66%, while there are only 128 physical and functional reasoning questions, accounting for about 11.7% of the total data, and most of them involve relatively simple applications of physical relationships or logical judgments. In contrast, actual optoelectronic scientific research usually requires comprehensive analysis combining multiple simulation results, judgment of physical mechanisms, and complex reasoning. Therefore, it remains questionable whether the existing tasks can fully represent the visual understanding and physical reasoning needs in real scientific research scenarios. This limitation weakens the representativeness of the benchmark for actual optoelectronic scientific research applications and the generalization value of its evaluation results.

  2. The experimental verification of the difficulty division and the analysis of physical reasoning ability is still insufficient. The authors divide task difficulty through preset construction parameters, but in the end only the difficulty parameters of 13 tasks passed experimental verification, indicating that the proposed difficulty division mechanism cannot effectively reflect differences in model capabilities on all tasks. In addition, the paper's core conclusion about the separation of visual understanding and physical reasoning ability is mainly based on 55 pairs of paired questions, among which there are only 21 optics-related pairs, so the experimental scale is relatively limited. At the same time, all tested models use closed or minimal reasoning configurations, lacking comparison of model performance under different reasoning intensities. Therefore, the existing experiments are not yet sufficient to fully verify the general effectiveness of the difficulty control mechanism, nor is it easy to determine whether the observed physical reasoning errors still exist under more sufficient reasoning settings. This limits the persuasiveness of the paper's core experimental conclusions.

Questions for authors
  1. Regarding the representativeness of the dataset and the coverage of complex physical reasoning tasks. At present, electrical-related questions account for about 66% of the dataset, while there are only 128 physical and functional reasoning questions, some of which mainly involve simple applications of physical relationships or logical judgments. Can the authors further explain the design rationale for the current task types and their distribution, and to what extent they can represent the visual understanding needs in actual optoelectronic scientific research? In particular, for tasks that require analyzing multiple simulation results or involve more complex physical mechanisms, can the current benchmark provide effective evaluation?

  2. Regarding the effectiveness of the difficulty division mechanism. The authors propose controlling task difficulty through preset construction parameters, but in the end only the difficulty parameters of 13 tasks passed experimental verification. For the remaining tasks, what is the main reason that the difficulty parameters failed to effectively distinguish model performance? Can the authors further analyze whether the difficulty division in these tasks is reasonable, and the applicability of this mechanism across different task types? This would help determine whether the controllable difficulty design can serve as a general advantage of this benchmark.

  3. Regarding the experimental support for the conclusion of insufficient physical reasoning ability. The paper analyzes the difference between visual understanding and physical reasoning through paired questions, but the related analysis is mainly based on 55 pairs of paired questions, among which there are only 21 optics-related pairs. Can the authors further explain the representativeness of these paired questions in physical reasoning tasks and provide more sufficient evidence to verify whether the observed reasoning errors are universal?

In addition, in the current experiments, all models have reasoning mode turned off or enabled at a minimal level. Can the authors supplement comparison results under different reasoning intensities, so as to clarify whether the physical reasoning errors mainly reflect the capability limitations of the models themselves or are significantly affected by the reasoning configuration?

4 comments

Response (W1 & Q1): Many thanks for the constructive comment. We acknowledge that the original manuscript did not clearly explain the design rationale for the current task types and their distribution. We would like to highlight that the Limitations (Sec. 4.3) already explain why physical and functional reasoning questions are fewer: each needs a physical relation that the expert confirms for that particular figure. Following the reviewer's suggestion, we explain the design rationale and the coverage of complex physical reasoning below.

Design rationale: every answer must be verifiable. Drawing scripts give exact ground truth at scale, whereas a physical and functional reasoning question needs expert confirmation; we chose a smaller set we can trust over a larger one with noisy gold answers. The share of electrical-related questions (about 66%) also reflects the domain: optoelectronic devices are driven and read out electrically, electronics has three levels to read (components, circuits, systems), and photonics has far fewer device classes. We will report the optical subset separately; it is the harder one for 5 of our 8 configurations.

Most items require domain-specific capabilities, not only the 11.7% on physical reasoning. We labelled each of the 39 task types by the capability it requires (the labels will be listed in a new appendix):

Capability requiredExampleItemsShare (%)
Optical figure understandingIdentify optical phenomena and their parameters in a figure42038
Domain-specific visual conventionsTell a NAND from an AND symbol50446
Domain-specific physical or functional reasoningInfer n_eff from mode confinement16815

In total, all 1,092 items evaluate optical figure understanding, with 672 items (62%) further requiring domain-specific visual conventions or physical reasoning.

The current benchmark evaluates the steps of complex physical reasoning. It already includes tasks that require analyzing multiple simulation results or several reasoning steps:

TaskItems
Compare multiple (2–4) simulation results135
L9: link structure, field and spectrum20
L8: reading plus calculation22

These steps are where models fail: even after a correct reading, the five reasoning-off models get the physical step right only 66% of the time (58% on the optics-related pairs), and the reasoning-enabled models only 76–85%. Longer chains in actual optoelectronic research would compound these errors, and an end-to-end benchmark would show that models fail but not which step fails. End-to-end scenarios with more complex physical mechanisms are the natural next step, as noted in the Limitations.


Response (W2 & Q2): We thank the reviewer for this insightful question. Following the reviewer's suggestion, we analyzed every task type below.

The main reasons are too few items per tier and a parameter that is not the bottleneck. On the five reasoning-off models, the difficulty parameters of 13 task types pass experimental verification; the remaining 25 fall into four groups:

OutcomeTask typesExample
Passed verification13Easy / medium / hard: 88 / 64 / 50
No difficulty parameter by design5Expert-written items
Too few items per tier10"Second-most confined": 3 items
Ceiling3Lane count: ≥ 97% in every tier
Parameter not the bottleneck8 (4 reversed)IC pins: larger packages are easier

In the "not the bottleneck" group, errors follow a factor that the parameter does not encode: for symbol recognition, how closely the wrong options resemble the gold matters more than how many of them share its family, and larger IC packages are easier because their pin rows are regular.

Applicability across task types: the mechanism works best for quantities. 10 of the 13 verified parameters are a comparison margin, a count or a required precision, and the error analysis agrees: 61–75% of the comparison errors of DeepSeek-V4.1, GPT-5.6-Terra (on) and GPT-5.6-Sol (on) fall in the smallest-margin tier, which holds 55% of the comparison items. We will add the per-type breakdown as an appendix table.

Response (W2 & Q3–Q4): Many thanks for these constructive suggestions. We acknowledge that the original manuscript did not sufficiently explain the representativeness of the paired questions, nor compare results under different reasoning intensities. We would like to highlight that the paired questions cover all three physical links that can be checked against simulation or drawing-script records, namely confinement → n_eff (11 pairs), index → resonance shift (10) and circuit → output (34) (Sec. 3.1), and that Sec. 4.3 already shows that 18 of the 20 wrong counterfactual answers move the resonance the wrong way after a correct reading. Following the reviewer's suggestion, we further tested the pairs statistically and added reasoning-enabled runs.

The reasoning errors are not specific to one model. The gap is significant for most models, on all pairs and on the optics-related pairs alike. For each model, we take the gap between descriptive and physical accuracy on the same pairs, then resample the pairs with replacement and recompute the gap each time. For most models, the gap stays above zero in 95% of the resamples:

Paired-question statisticAll 55 pairs21 optical pairs
Models whose gap stays above zero in 95% of resamples4 / 55 / 5
Pooled P(phys. correct | desc. correct)66%58%

Moreover, when the correct reading is given as text, every model solves the mode → n_eff step (100%, vs. 45–91% from the figure), so the errors lie in linking the figure to the physics rather than in physics knowledge.

The physical reasoning errors persist under more sufficient reasoning settings. Beyond reasoning turned off or enabled at a minimal level, we evaluated all 1,092 items with reasoning enabled for GPT-5.6-Terra, GPT-5.6-Sol and Qwen3-VL-235B-Thinking:

Accuracy (%)Terra (off)Terra (on)Sol (on)Qwen3-VL (off)Qwen3-VL (on)
Overall67.382.488.571.878.4
Count peaks5662692562
Which spectrum has more peaks861001007395
P(phys. correct | desc. correct), 55 pairs5276847985
P(phys. correct | desc. correct), 21 optics-related pairs53769063100

Enabling reasoning helps significantly (Terra: McNemar p < 1e-25; Qwen3-VL: p < 1e-6), but after a correct reading the reasoning-enabled models still fail the physical step 15–24% of the time, the gap CI still excludes zero for Terra (on) and Sol (on), and peak counting stays weak (62–69%). These errors therefore reflect the capability limitations of the models themselves, not only the reasoning configuration. We will add the three reasoning-enabled runs to the main table.

We sincerely hope these responses address the reviewer's concerns, and we would be grateful if the reviewer could reconsider the rating in light of them. We are happy to answer any further questions.

Thank you for the detailed response and additional analyses. They have addressed some of my main concerns, particularly regarding the reasoning settings and difficulty analysis. I have therefore raised my score to 3.

Thank you for taking the time to consider our rebuttal and update your assessment. We are glad that the reasoning-enabled evaluations and the analysis of difficulty across task types addressed some of your main concerns. Your feedback helped us improve both parts of the evaluation, and we will incorporate these results into the revised manuscript.

Reviewer_hgpGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents OptoFig, a benchmark for evaluating vision-language models on optoelectronic device figures. The benchmark consists of 1,092 questions over 799 figures, covering 39 task types organized into three categories. The authors construct the dataset using device simulations, programmatically generated circuit drawings, and expert annotations. Experiments on seven VLMs show overall accuracies ranging from 54.2% to 74.5%. Through paired descriptive questions, the authors identify two major limitations: models struggle with precise visual measurements, and correct visual perception does not necessarily lead to correct physical reasoning.

Critical strengths
  • Meaningful Application Domain. Understanding scientific figures is an important capability for AI-assisted research. OptoFig focuses on optoelectronic device figures, covering both device structures and performance characterization, which complements existing scientific-figure and engineering benchmarks.

  • Systematic Data Construction. The benchmark uses simulation-grounded answers, automated checks, and expert verification to improve annotation reliability.

Critical weaknesses

Many questions are generated from predefined templates and variations of the same simulation projects, raising concerns about their effective diversity. Additionally, 721 of 1,092 questions concern electrical figures, while only 128 evaluate physical and functional reasoning.

The claimed perception–reasoning gap is primarily supported by a small number of paired questions. Moreover, all models are evaluated with reasoning disabled or minimized, without considering external tools or agentic workflows.

Several closely related benchmarks, such as CircuitVQA and ChartMuseum, are not sufficiently discussed. CircuitVQA evaluates circuit recognition and counting, while ChartMuseum investigates visual reasoning limitations through controlled experiments. The paper should better clarify its relationship to these works and the already cited CircuitSense.

Questions for authors
  • Q1: Could the authors provide a breakdown of the 799 figures by image type and simulation project. How many distinct question templates are used?

  • Q2: Could the authors conduct independent human evaluation on a subset with high initial error rates to assess visual ambiguity and report human accuracy? Additionally, evaluating reasoning-enabled models and tool-augmented agents would help determine whether the observed limitations persist in realistic settings.

  • Q3: The ±10% tolerance may be too strict for plot-based estimates yet too permissive for numerical calculations. Could the authors evaluate sensitivity to task-specific tolerances, quantify errors in automated answer extraction, and compare multiple-choice with free-response performance on convertible questions to assess potential answer-choice bias?

  • Q4: Could the authors clarify the definitions of "Controlled difficulty" and "Simulated performance" in Table 1, particularly regarding CircuitSense and QCalEval? How do these criteria distinguish OptoFig from the corresponding capabilities in prior benchmarks?

7 comments

Response (W1 & Q1): Many thanks for the constructive comment. We acknowledge that the original manuscript did not provide a breakdown of the 799 figures by image type and simulation project. Following the reviewer's suggestion, we provide the breakdown and the number of templates below, and show that the questions are diverse.

Breakdown of the 799 figures (Q1). The 253 optical figures come from 57 simulation projects, and the 546 electrical figures from 9 drawing-script categories:

Image typeFigures
Optical, from 57 simulation projects253
Spectra or curves85
Field maps82
Mode profiles49
Device structures29
Others8
Electrical, from 9 drawing-script categories546
Schematics391
Timing diagrams94
Packages or structures42
Others19

Distinct question templates (Q1). We use 39 question templates, one per task type; after rewording, the questions have 279 distinct stems.

Effective diversity. The questions are not concentrated on a few figures, simulation projects or circuit contents:

Diversity statisticValue
Number of figures used by only one question530 of 799
Number of optical questions from the single simulation project with the most questions46 of 371 (12%)
Median number of optical questions per simulation project4
Number of simulation projects with ≤ 5 questions33 of 57
Number of distinct component classes asked about (in 19 families)67
Number of distinct correct answers to component-symbol questions83
Number of distinct gate types among gate-identity answers8
Number of distinct IC pin counts (range 2–32)13
Number of distinct Boolean functions among 39 multi-gate circuits25

The electrical figures are script-generated by design, so that every answer is traceable to the drawing script. Even so, questions from the same template are not near-duplicates: the accuracy of a single model ranges from 25% to 100% across component families for DeepSeek-V4.1 (35% to 100% for DeepSeek-V4).

Why 721 of 1,092 questions concern electrical figures. The share reflects the domain: (1) electronics has more levels to read (components, circuits and whole systems), each with its own figures; (2) photonic devices are driven and read out electrically, so reading an optoelectronic figure presupposes reading its electrical side; and (3) photonics has far fewer device classes, since the field is less mature. Even so, the optical subset is the harder one for 5 of our 8 configurations (e.g., DeepSeek-V4.1: 70.6% optical vs. 76.6% electrical), and we will report both subsets separately.

Only 128 questions evaluate physical and functional reasoning, but 672 require domain knowledge. The 128 count only questions whose final step is physical. By the capability each item requires, 672 items (62%) need domain knowledge: 504 need domain-specific visual conventions and 168 need physical or functional reasoning.

Response (W2): We thank the reviewer for this insightful comment. Following the reviewer's suggestion, we add three sets of experiments below: significance tests on the paired questions, a full evaluation with reasoning enabled, and a test with an external zoom tool.

The gap is significant despite the small number of paired questions. For each model, we take the gap between descriptive and physical accuracy on the same pairs, then resample the pairs with replacement and recompute the gap each time. For most models, the gap stays above zero in 95% of the resamples:

Paired-question statisticAll 55 pairs21 optical pairs
Models whose gap stays above zero in 95% of resamples4 / 55 / 5
Pooled P(phys. correct | desc. correct)66%58%

Moreover, when the correct reading is given as text, every model solves the mode → n_eff step (100%, vs. 45–91% from the figure), so the failure lies in linking the figure to the physics.

The gap persists with reasoning enabled. Beyond the setting with reasoning disabled or minimized, we evaluated all 1,092 items with reasoning enabled for GPT-5.6-Terra, GPT-5.6-Sol and Qwen3-VL-235B-Thinking:

Accuracy (%)Terra (off)Terra (on)Sol (on)Qwen3-VL (off)Qwen3-VL (on)
Overall67.382.488.571.878.4
Count peaks5662692562
Which spectrum has more peaks861001007395
P(phys. correct | desc. correct), 55 pairs5276847985
P(phys. correct | desc. correct), 21 optical pairs53769063100

Reasoning helps overall (Terra fixes 212 items and breaks 47), most on quantitative reading (+26 points) and less on the physical step (+18). After a correct reading, the reasoning-enabled models still fail the physical step 15–24% of the time. We will add the three reasoning-enabled runs to the main table.

An external tool does not fix peak counting. As a minimal test, we cropped and enlarged the counting figures, as a zoom tool would (accuracy, %):

Task, configurationOriginalCropped
Gate inputs, Terra (on)3367
Gate inputs, Sol (on)7088
Peaks, Terra (on)6250
Peaks, Sol (on)6969

Zoom helps with dense symbols such as gate inputs but not with peaks, which need structured extraction. For agentic workflows, OptoFig isolates the figure-reading step they depend on: if a single step fails, chaining steps compounds the failure.


Response (W3): Thank you for pointing us to these closely related benchmarks. We acknowledge that the original manuscript did not sufficiently discuss CircuitVQA and ChartMuseum. Following the reviewer's suggestion, we will add CircuitVQA and ChartMuseum to Table 1 and the Related Work section.

  • CircuitVQA evaluates circuit recognition and counting on schematics, including hand-drawn ones; it is closest to our symbol-recognition and counting task types, but contains no simulated device performance or paired reasoning questions.
  • ChartMuseum investigates visual reasoning limitations through controlled experiments on a separate synthetic set, finding that accuracy falls as visual complexity grows; this is consistent with our finding that models perceive figures with limited precision. OptoFig instead sets the difficulty of each item within the benchmark itself.
  • CircuitSense tests schematic perception, analysis and design, but contains no simulated device performance.

[R1] Rahul Mehta et al. CircuitVQA: A Visual Question Answering Dataset for Electrical Circuit Images. ECML PKDD 2024.

[R2] Liyan Tang et al. ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models. NeurIPS 2025.

Response (Q2): We thank the reviewer for this valuable suggestion. We would like to highlight that every item passed expert review, and that after a first evaluation round, 109 flagged items, including 68 on which at least six of eight VLMs gave the same wrong answer, were returned to the expert for a second review, which corrected 11 and removed 45 (Sec. 3.4, Appendix E). Following the reviewer's suggestion, we further audited model errors for visual ambiguity and will report the full human review process.

Visual ambiguity. Auditing 50 sampled errors of Terra (on), we found one genuine ambiguity: in some "first rising lane" timing items, the gold lane is already high at the left edge, while models answer with the lane that shows the first rising edge. We will reword these items to "the first lane that is high and stays high" and re-score them. In addition, the model itself corrects 13 of the 50 errors when asked to explain its answer, so these errors do not stem from visual ambiguity.

Human review. The subset with high initial error rates has already been reviewed by a human. Our domain expert first confirmed, corrected or dropped every item: 1,016 items individually, and the remaining 76 items of the most uniform task type after a random check of 27 found no errors. After the first evaluation round, the expert then re-examined the 109 flagged items, as described above. We will report this process and its outcomes in detail in the revised manuscript.

Reasoning-enabled models and tool-augmented agents. The main observed limitations persist in these settings (see W2): with reasoning enabled, the physical step still fails 15–24% of the time after a correct reading, and a zoom tool does not improve peak counting.


Response (Q3): Many thanks for these careful suggestions. Following the reviewer's suggestions, we conducted all three analyses.

±10% is not too strict for plot-based estimates, but too permissive for L8. Overall accuracy of the five reasoning-off models (%):

ToleranceDeepSeek-V4.1DeepSeek-V4Qwen3-VLTerraGPT-4o
±1%73.271.270.066.265.8
±5%74.172.270.967.166.8
±10%74.572.671.867.367.8
±20%75.173.271.967.868.1

Accuracy moves by at most 2.3 points, and loosening ±10% to ±20% adds at most 0.6 points; the ranking changes only where Terra and GPT-4o, 0.5 points apart at ±10%, swap places. For the numerical calculation in L8, however, the correct shift is only 4–11% of the wavelength, so an unshifted answer passes on all 10 items. We will therefore use a task-specific tolerance of ±1% for L8, which strengthens the reasoning-gap finding (% of the 10 items):

Model±10%±1%Moved the right way
DeepSeek-V4.1502030
DeepSeek-V4501030
Qwen3-VL80050
Terra402050
GPT-4o80050

Automated answer extraction makes no errors. Only 30 of 8,736 replies (all from DeepSeek) lack a JSON answer and fall back to a regex. Re-grading them by the model's last stated answer agrees in all 28 cases that have one; the other 2 hit the token limit and are wrong either way.

Removing the options does not change the ranking at either end. Sol (on) stays first, and DeepSeek-V4 and GPT-4o stay in the last two places. We removed the options from the 112 convertible questions:

Task types (items)Accuracy drop without options (points)
Gate identity (37)−11 to +8
Plotted quantity and figure type (75)27–56

On gate identity, the effect has no consistent direction, so we find no systematic answer-choice bias there. The larger drop on plotted quantity and figure type partly reflects that free answers name the same thing differently; lenient scoring recovers about 7 points. We will report the free-response variant for these task types.

Reviewer_vcYtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces OptoFig, a benchmark of 1,092 questions over 799 optoelectronic device figures covering perception, quantitative analysis, and physical reasoning. It provides traceable reference answers, construction-based difficulty levels, and paired questions to examine figure reading and subsequent reasoning separately. Validation includes automated checks, expert review, and evaluations of seven VLMs with reasoning disabled or minimized.

Critical strengths
  1. The paper evaluates both design and characterization figures for optoelectronic devices, covering device structures, circuit diagrams, field maps, spectra, and timing diagrams. These figures are directly relevant to device design and analysis, giving the benchmark a clear scientific motivation and complementing general-purpose figure-understanding benchmarks.

  2. Reference answers are traceable to image measurements, simulation or drawing records, and expert annotations. Automated checks, text-only testing, and expert review support data quality. Paired descriptive and reasoning questions, together with analyses of counting errors and incorrect resonance shifts, help distinguish difficulties in reading figures from difficulties in applying physical relationships.

Critical weaknesses
  1. The paper provides limited insight into model failures. The analysis identifies several error patterns but does not systematically examine the strongest models’ failures across the main tasks. It remains unclear why these failures occur and which underlying limitations future methods should address.

  2. The benchmark’s added evaluation value is unclear. Most questions focus on perception and quantitative figure reading, while physical and functional reasoning accounts for only about 12%. The paper does not sufficiently establish what additional domain-specific capabilities it evaluates beyond existing scientific figure and circuit benchmarks.

Questions for authors

Please see the Critical Weaknesses section for my main concerns.

4 comments

Response (W1): Many thanks for the constructive comment. We acknowledge that the original manuscript did not systematically examine the strongest models' failures across the main tasks. We would like to highlight that Sec. 4.3 identifies two error patterns that partly explain why these failures occur: most wrong peak counts are near misses, and 18 of the 20 wrong counterfactual answers move the resonance the wrong way after a correct reading. Following the reviewer's suggestion, we further examined the errors of the strongest reasoning-off model (DeepSeek-V4.1) and of the two strongest reasoning-enabled configurations (GPT-5.6-Terra and GPT-5.6-Sol, marked "on" below), attributing each error from construction metadata rather than from model self-report:

Error statistic% of errors
Peak count off by exactly one74
Peak count too low78
Gate-input count too low100
Comparison error in the smallest-margin tier (55% of items)61–75

Most recognition errors of the two reasoning-enabled configurations pick a look-alike wrong option from the same family (e.g., NAND for AND), whereas those of DeepSeek-V4.1 split evenly:

Wrong recognition optionDeepSeek-V4.1Terra (on)Sol (on)
Same family as the gold25239
Other family2794

To identify why these failures occur, we changed one factor at a time. First, cropping and enlarging the 163 counting figures helps with dense symbols but not with peaks (accuracy, %):

Counting taskTerra (off)Terra (on)Sol (on)
All 163 figures44 → 7275 → 8585 → 87
Gate inputs18 → 3333 → 6770 → 88
Peaks only56 → 5662 → 5069 → 69

Second, models that fail on the figure largely succeed once the same content is given as text (accuracy, %):

Text givenTaskModelFigure → text
NetlistCircuit output (39 circuits)DeepSeek-V474 → 100
NetlistCircuit output (39 circuits)Terra (off)74 → 92
Correct readingMode → n_effEvery model45–91 → 100

The results show that these failures stem from three underlying limitations that future methods should address: (i) input resolution for dense symbols such as gate inputs, which cropping partly addresses; (ii) exact counting of repeated features such as peaks, which cropping does not fix; and (iii) reasoning from the figure to the underlying physics or circuit logic, which models largely manage once the netlist or the reading is given as text. We will add this analysis as a new section in the revised manuscript.

Response (W2): We thank the reviewer for this insightful comment. We acknowledge that the original manuscript did not sufficiently clarify what additional domain-specific capabilities OptoFig evaluates beyond existing scientific figure benchmarks. We would like to highlight that OptoFig evaluates three domain-specific capabilities that, to our knowledge, no existing scientific figure or circuit benchmark covers: reading simulator outputs of devices, with answers taken from the simulation record; applying the physical links between these outputs, such as confinement → effective index; and turning a correct reading into a correct physical conclusion, which the paired questions on the same figure test (Table 1). Following the reviewer's suggestion, we further quantified these capabilities and compared OptoFig with an existing scientific figure benchmark.

Most items require domain-specific capabilities, not only the 12% on physical reasoning. We labelled each of the 39 task types by the capability it requires (the labels will be listed in a new appendix):

Capability requiredExampleItemsShare (%)
Optical figure understandingIdentify optical phenomena and their parameters in a figure42038
Domain-specific visual conventionsTell a NAND from an AND symbol50446
Domain-specific physical or functional reasoningInfer n_eff from mode confinement16815

In total, all 1,092 items evaluate optical figure understanding, with 672 items (62%) further requiring domain-specific visual conventions or physical reasoning.

OptoFig reveals differences that an existing scientific figure benchmark does not. We ran CharXiv-reasoning (300 random questions) under our protocol on seven configurations (five reasoning-off models, plus Terra and Sol with reasoning enabled). The rankings only partly agree (Spearman 0.82 over the seven configurations, 0.60 over the five reasoning-off models), and models that are close on CharXiv differ sharply on OptoFig (accuracy, %):

ModelCharXivOptoFig: gate inputsOptoFig: IC pinsOptoFig: P(phys. correct | desc. correct), optical pairs
Terra (on)75.7339076
Sol (on)73.3709490
Terra (off)57.0184353
Qwen3-VL56.7558063

For example, Terra (off) and Qwen3-VL score 57.0 and 56.7 on CharXiv, but 18% and 55% on gate inputs, and 43% and 80% on IC pins. This is the added evaluation value of OptoFig: beyond ranking models, it shows which domain-specific step fails.

We sincerely hope these responses address the reviewer's concerns, and we would be grateful if the reviewer could reconsider the rating in light of them. We are happy to answer any further questions.

Thank you for the detailed response. It addresses most of my concerns and I will raise my score.

Thank you for taking the time to read our rebuttal and update your assessment. We are glad that the expanded error analysis and the clarification of what OptoFig evaluates addressed most of your concerns. Your feedback helped us sharpen both points, and we will incorporate these additions into the revised manuscript.

Comments on the paper

1 comment
Area_Chair_BFUiOpenReview ↗

Urgent Reminder: Discussion Period Ends November 18

Dear Reviewers,

The discussion period will end on November 18, 2026. Please carefully review the authors' responses and actively engage in discussions.

If you have not yet initiated a discussion, please do so immediately. Reviewers are expected to address the authors' rebuttals, provide constructive follow-up comments, and update their evaluations when appropriate.

Please also ensure that your reviews reflect independent human judgment and that any AI assistance is properly documented in accordance with the review guidelines.

Active participation is an essential reviewer responsibility and is critical to ensuring a rigorous and fair review process.

Thank you for your cooperation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.