- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a vision-language model (VLM) evaluation benchmark for optoelectronic device image understanding, aiming to evaluate the ability of existing VLMs to understand device design images, performance characterization images, and related physical information. The dataset contains 799 images and 1,092 question-answer samples, covers 39 task types, and is mainly divided into three categories: visual perception and recognition, quantitative analysis and comparison, and physical and functional reasoning. The authors use simulation data, drawing scripts, and domain expert review to construct standard answers, and divide task difficulty through preset construction parameters, so as to improve the reliability and interpretability of the evaluation.
In terms of experiments, the authors evaluate seven mainstream VLMs, with overall accuracy ranging from 54.2% to 74.5%. Through the analysis of different task types, difficulty levels, and paired questions, the authors point out that existing models have two main limitations: first, they can recognize the overall features of images, but perform insufficiently in precise counting and numerical reading; second, even if they correctly read the image information, they may still fail to further draw correct physical conclusions.
Overall, this paper provides a dedicated benchmark of certain value for evaluating the image understanding ability of VLMs in the optoelectronics field. However, the dataset still has limitations in task distribution and coverage of complex physical reasoning, the effectiveness of the difficulty division has not been fully verified, and the experimental support for conclusions related to physical reasoning is also relatively limited. Therefore, the current experimental results can reflect the shortcomings of models on specific tasks, but are not yet sufficient to fully prove the representativeness of this benchmark for broader optoelectronic scientific research scenarios.
Critical strengths
-
A domain-targeted evaluation benchmark that can assess different capabilities of VLMs in optoelectronic image understanding. This paper constructs an evaluation benchmark specifically for optoelectronic devices, covering multiple image types such as device structures, optical field distributions, spectra, and circuit diagrams, and divides 39 tasks into three categories: visual perception and recognition, quantitative analysis and comparison, and physical and functional reasoning. Compared with general scientific image understanding benchmarks, this dataset has clearer domain specificity and can provide a reference for analyzing differences in VLM capabilities on optoelectronics-related tasks.
-
The dataset construction and validation process is relatively standardized, and it provides experimental analysis with certain interpretability. The authors establish standard answers by combining simulation results, drawing scripts, and domain expert review, reducing the risk of errors caused by relying entirely on automatic model-generated annotations. At the same time, they divide task difficulty through preset construction parameters and use paired questions to analyze the difference between visual understanding and subsequent physical reasoning. The related experiments not only report the overall accuracy of the models but also reveal the insufficient precise counting ability and the phenomenon that physical reasoning errors may still occur after correctly reading images. These analyses help to understand the capability limitations of existing VLMs in a more detailed way.
Critical weaknesses
-
The representativeness of the dataset and the coverage of tasks are limited, especially the lack of sufficient complex physical reasoning tasks. Although OptoFig covers a variety of optoelectronics-related images and 39 task types, the task distribution of the dataset is obviously unbalanced. Among them, electrical-related questions account for about 66%, while there are only 128 physical and functional reasoning questions, accounting for about 11.7% of the total data, and most of them involve relatively simple applications of physical relationships or logical judgments. In contrast, actual optoelectronic scientific research usually requires comprehensive analysis combining multiple simulation results, judgment of physical mechanisms, and complex reasoning. Therefore, it remains questionable whether the existing tasks can fully represent the visual understanding and physical reasoning needs in real scientific research scenarios. This limitation weakens the representativeness of the benchmark for actual optoelectronic scientific research applications and the generalization value of its evaluation results.
-
The experimental verification of the difficulty division and the analysis of physical reasoning ability is still insufficient. The authors divide task difficulty through preset construction parameters, but in the end only the difficulty parameters of 13 tasks passed experimental verification, indicating that the proposed difficulty division mechanism cannot effectively reflect differences in model capabilities on all tasks. In addition, the paper's core conclusion about the separation of visual understanding and physical reasoning ability is mainly based on 55 pairs of paired questions, among which there are only 21 optics-related pairs, so the experimental scale is relatively limited. At the same time, all tested models use closed or minimal reasoning configurations, lacking comparison of model performance under different reasoning intensities. Therefore, the existing experiments are not yet sufficient to fully verify the general effectiveness of the difficulty control mechanism, nor is it easy to determine whether the observed physical reasoning errors still exist under more sufficient reasoning settings. This limits the persuasiveness of the paper's core experimental conclusions.
Questions for authors
-
Regarding the representativeness of the dataset and the coverage of complex physical reasoning tasks. At present, electrical-related questions account for about 66% of the dataset, while there are only 128 physical and functional reasoning questions, some of which mainly involve simple applications of physical relationships or logical judgments. Can the authors further explain the design rationale for the current task types and their distribution, and to what extent they can represent the visual understanding needs in actual optoelectronic scientific research? In particular, for tasks that require analyzing multiple simulation results or involve more complex physical mechanisms, can the current benchmark provide effective evaluation?
-
Regarding the effectiveness of the difficulty division mechanism. The authors propose controlling task difficulty through preset construction parameters, but in the end only the difficulty parameters of 13 tasks passed experimental verification. For the remaining tasks, what is the main reason that the difficulty parameters failed to effectively distinguish model performance? Can the authors further analyze whether the difficulty division in these tasks is reasonable, and the applicability of this mechanism across different task types? This would help determine whether the controllable difficulty design can serve as a general advantage of this benchmark.
-
Regarding the experimental support for the conclusion of insufficient physical reasoning ability. The paper analyzes the difference between visual understanding and physical reasoning through paired questions, but the related analysis is mainly based on 55 pairs of paired questions, among which there are only 21 optics-related pairs. Can the authors further explain the representativeness of these paired questions in physical reasoning tasks and provide more sufficient evidence to verify whether the observed reasoning errors are universal?
In addition, in the current experiments, all models have reasoning mode turned off or enabled at a minimal level. Can the authors supplement comparison results under different reasoning intensities, so as to clarify whether the physical reasoning errors mainly reflect the capability limitations of the models themselves or are significantly affected by the reasoning configuration?
4 comments
Response (W2 & Q3–Q4): Many thanks for these constructive suggestions. We acknowledge that the original manuscript did not sufficiently explain the representativeness of the paired questions, nor compare results under different reasoning intensities. We would like to highlight that the paired questions cover all three physical links that can be checked against simulation or drawing-script records, namely confinement → n_eff (11 pairs), index → resonance shift (10) and circuit → output (34) (Sec. 3.1), and that Sec. 4.3 already shows that 18 of the 20 wrong counterfactual answers move the resonance the wrong way after a correct reading. Following the reviewer's suggestion, we further tested the pairs statistically and added reasoning-enabled runs.
The reasoning errors are not specific to one model. The gap is significant for most models, on all pairs and on the optics-related pairs alike. For each model, we take the gap between descriptive and physical accuracy on the same pairs, then resample the pairs with replacement and recompute the gap each time. For most models, the gap stays above zero in 95% of the resamples:
| Paired-question statistic | All 55 pairs | 21 optical pairs |
|---|---|---|
| Models whose gap stays above zero in 95% of resamples | 4 / 5 | 5 / 5 |
| Pooled P(phys. correct | desc. correct) | 66% | 58% |
Moreover, when the correct reading is given as text, every model solves the mode → n_eff step (100%, vs. 45–91% from the figure), so the errors lie in linking the figure to the physics rather than in physics knowledge.
The physical reasoning errors persist under more sufficient reasoning settings. Beyond reasoning turned off or enabled at a minimal level, we evaluated all 1,092 items with reasoning enabled for GPT-5.6-Terra, GPT-5.6-Sol and Qwen3-VL-235B-Thinking:
| Accuracy (%) | Terra (off) | Terra (on) | Sol (on) | Qwen3-VL (off) | Qwen3-VL (on) |
|---|---|---|---|---|---|
| Overall | 67.3 | 82.4 | 88.5 | 71.8 | 78.4 |
| Count peaks | 56 | 62 | 69 | 25 | 62 |
| Which spectrum has more peaks | 86 | 100 | 100 | 73 | 95 |
| P(phys. correct | desc. correct), 55 pairs | 52 | 76 | 84 | 79 | 85 |
| P(phys. correct | desc. correct), 21 optics-related pairs | 53 | 76 | 90 | 63 | 100 |
Enabling reasoning helps significantly (Terra: McNemar p < 1e-25; Qwen3-VL: p < 1e-6), but after a correct reading the reasoning-enabled models still fail the physical step 15–24% of the time, the gap CI still excludes zero for Terra (on) and Sol (on), and peak counting stays weak (62–69%). These errors therefore reflect the capability limitations of the models themselves, not only the reasoning configuration. We will add the three reasoning-enabled runs to the main table.
We sincerely hope these responses address the reviewer's concerns, and we would be grateful if the reviewer could reconsider the rating in light of them. We are happy to answer any further questions.
Thank you for the detailed response and additional analyses. They have addressed some of my main concerns, particularly regarding the reasoning settings and difficulty analysis. I have therefore raised my score to 3.
Thank you for taking the time to consider our rebuttal and update your assessment. We are glad that the reasoning-enabled evaluations and the analysis of difficulty across task types addressed some of your main concerns. Your feedback helped us improve both parts of the evaluation, and we will incorporate these results into the revised manuscript.
Response (W1 & Q1): Many thanks for the constructive comment. We acknowledge that the original manuscript did not clearly explain the design rationale for the current task types and their distribution. We would like to highlight that the Limitations (Sec. 4.3) already explain why physical and functional reasoning questions are fewer: each needs a physical relation that the expert confirms for that particular figure. Following the reviewer's suggestion, we explain the design rationale and the coverage of complex physical reasoning below.
Design rationale: every answer must be verifiable. Drawing scripts give exact ground truth at scale, whereas a physical and functional reasoning question needs expert confirmation; we chose a smaller set we can trust over a larger one with noisy gold answers. The share of electrical-related questions (about 66%) also reflects the domain: optoelectronic devices are driven and read out electrically, electronics has three levels to read (components, circuits, systems), and photonics has far fewer device classes. We will report the optical subset separately; it is the harder one for 5 of our 8 configurations.
Most items require domain-specific capabilities, not only the 11.7% on physical reasoning. We labelled each of the 39 task types by the capability it requires (the labels will be listed in a new appendix):
In total, all 1,092 items evaluate optical figure understanding, with 672 items (62%) further requiring domain-specific visual conventions or physical reasoning.
The current benchmark evaluates the steps of complex physical reasoning. It already includes tasks that require analyzing multiple simulation results or several reasoning steps:
These steps are where models fail: even after a correct reading, the five reasoning-off models get the physical step right only 66% of the time (58% on the optics-related pairs), and the reasoning-enabled models only 76–85%. Longer chains in actual optoelectronic research would compound these errors, and an end-to-end benchmark would show that models fail but not which step fails. End-to-end scenarios with more complex physical mechanisms are the natural next step, as noted in the Limitations.
Response (W2 & Q2): We thank the reviewer for this insightful question. Following the reviewer's suggestion, we analyzed every task type below.
The main reasons are too few items per tier and a parameter that is not the bottleneck. On the five reasoning-off models, the difficulty parameters of 13 task types pass experimental verification; the remaining 25 fall into four groups:
In the "not the bottleneck" group, errors follow a factor that the parameter does not encode: for symbol recognition, how closely the wrong options resemble the gold matters more than how many of them share its family, and larger IC packages are easier because their pin rows are regular.
Applicability across task types: the mechanism works best for quantities. 10 of the 13 verified parameters are a comparison margin, a count or a required precision, and the error analysis agrees: 61–75% of the comparison errors of DeepSeek-V4.1, GPT-5.6-Terra (on) and GPT-5.6-Sol (on) fall in the smallest-margin tier, which holds 55% of the comparison items. We will add the per-type breakdown as an appendix table.