acceptodds
Under review as a conference paper at ICLR 2027

SEM-VQA: Structured Visual Question Answering for Scientific Understanding of Scanning Electron Microscope Images

Abstract

Understanding scanning electron microscope (SEM) images is central to materials science, yet progress on applying multimodal large language models (MLLMs) to this domain is limited by the scarcity of large-scale, open-ended visual question answering (VQA) datasets grounded in real scientific imagery. We introduce SEM-VQA, a pipeline for constructing and evaluating SEM-focused VQA data that enforces image-grounded reasoning. Using LLM-assisted annotation and prompting constraints requiring answers to be derivable from the image alone, we build a corpus of 224,026 question-answer-visual-evidence triplets over 47,981 SEM micrographs, organised into four levels of visual understanding: observation, detection, identification, and interpretation. We construct a held-out benchmark of 300 images via embedding-based clustering and use it to evaluate open-source and proprietary MLLMs under an LLM-as-judge protocol. Fine-tuning a compact 2B-parameter Qwen3.5 model on our corpus raises its judged score from 3.33 to 3.83 (5-point scale), a 14.9% relative gain that closes roughly 76% of the gap to GPT-5.4-mini (3.98). The fine-tuned model significantly outperforms Claude-Haiku-4.5 and, by a narrower margin, Gemini-3.1-Flash, though a modest but significant deficit remains relative to GPT-5.4-mini; it also leads all evaluated systems across every automatic metric (token-F1, BERTScore-F1, STS, ROUGE-L, METEOR, BLEU). These results show that targeted, structured dataset construction, rather than model scale alone, can substantially close the gap between small open models and large proprietary systems on specialised scientific visual reasoning. We release our dataset, benchmark, and evaluation protocol.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
visual question answering, multimodal large language models, scanning electron microscopy, datasets and benchmarks, scientific image understanding, vision-language models, Domain-specific AI, open end VQA

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BCLbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SEM-VQA, a large VQA dataset for scientific understanding of SEM images. The dataset contains 224,026 question–answer–visual-evidence triplets derived from 47,981 SEM images collected from materials-science publications. Questions are generated using Gemini-2.5-Flash with image, caption, and summary inputs and organized into four levels: Observation, Detection, Identification, and Interpretation. The authors construct a held-out benchmark of 300 images and QA pairs using embedding-based clustering and evaluate multiple open-weight and proprietary multimodal language models using six automatic metrics and an LLM-based judge.

Critical strengths

S1. Large-scale dataset with structured question categories. The benchmark is much larger than existing materials-specific VQA benchmarks. The four-level taxonomy provides a useful framework for organizing questions about visual observation, feature detection, technical identification, and scientific interpretation. The inclusion of visual-evidence descriptions is also potentially valuable for developing interpretable scientific VQA systems.

Critical weaknesses

W1. My concern is about the reliability of the benchmark annotations. The entire corpus is generated using a single MLLM, and it is unclear how human evaluation is involved. Given the specialized nature of SEM interpretation, automatically generated answers may contain scientific inaccuracies.

W2. Limited size and validation of the held-out benchmark. Although the training corpus is large, the evaluation benchmark contains only 300 images. It would be better to report performance separately for the four question categories in Table 1.

W3. There are some inconsistencies. The main paper repeatedly reports 47,981 images, whereas the reproducibility statement on page 10 says 16,192 images. The abstract also refers to GPT-5.4-mini, while the main experimental tables identify GPT-5-mini.

Questions for authors

W2 and W3 in the above weakness section.

Reviewer_4DjLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces SEM-VQA, a visual question-answering corpus for scientific interpretation of scanning electron microscope images, containing 224,026 triplets over 47,981 micrographs, organized into four prompt-defined levels. The authors then construct rep300, a held-out benchmark of 300 images using embedding-based stratification. They also fine-tune a 2B-parameter model. The experiments show substantial in-domain gains after fine-tuning, while also demonstrating that conventional text-overlap metrics do not reliably reproduce the LLM-judge ranking.

Critical strengths

SEM-VQA breaks through the limitations of most existing datasets, which are mostly small in scale and use multiple-choice questions. It organizes a large-scale dataset of SEM image question-answer pairs in four levels, and provides detailed annotations.

Critical weaknesses

(1) The selected generator was evaluated only by one annotator on 34 images and 151 pairs of generated images, and the complete 224,026 pairs of the corpus and 1,497 pairs of the rep300 benchmark were not validated.

(2) The VQA annotations were generated using Gemini-2.5-Flash. However, Gemini-3.5-Flas, which serves as the judge, cannot view the SEM images. Therefore, it can measure the consistency with the synthetic reference through questions, answers, and results, but it cannot determine whether the reference or prediction is supported by the images.

(3) The main paper contains no figures, while the appendix figures primarily visualize the small generator-selection pilot. The paper does not show representative SEM images paired with captions, contextual summaries, questions from each level, reference answers, model predictions, and judge decisions.

Questions for authors

(1) How did the authors verify that the benchmark questions are answerable from the SEM image alone, especially for Identification and Interpretation questions whose terminology may be supported by the caption or contextual summary? What fraction of rep300 has been independently checked by experts under image-only conditions?

(2) Could the authors include representative SEM images together with their captions, contextual summaries, questions from each taxonomy level, reference answers, and model predictions?

(3) Could the authors reconcile the 47,981 images reported in the abstract and dataset section with the 16,192 images stated in the Reproducibility Statement? Please also clarify why rep300 is introduced as containing 1,497 question–answer pairs, while the main model comparison uses 1,492 pairs and the common valid subset contains 1,472 items.

(4) Appendix A defines the four levels and gives several short prompt examples, but the paper does not present actual generated records containing the SEM image, question, answer, level, feature, and visual-evidence fields. Could the authors provide several complete examples from each level, including borderline or rejected cases?

(5) The prompt recommends questions such as “Which region contains the most porosity?” and “Which area shows rougher surface texture?”. How are regions defined when the image contains no coordinates, panel labels, segmentation masks, or standardized spatial divisions? Are descriptions such as “upper left” or “central region” required?

1 comment

We thank the reviewer for the detailed review. We have uploaded a revised PDF; changes are summarised below.Kindly reconsider your score

W3 / Q2 / Q4 Figures and complete examples. We agree that this was the paper's main presentation gap, and we have fixed it.

  • A new figure shows a complete rep300 record: SEM image, sub-caption, summary, every question with its level, reference answer, zero-shot and fine-tuned predictions, and judge scores.
  • A new appendix, "Complete benchmark records", contains that figure and gives 8 complete records with every field, including the judge's verdict and justification. These are the main-figure image, one typical image per visual cluster, and the two images where fine-tuning lowered the score most.
  • Images are selected by a fixed rule, not by hand: an image is "typical" when its zero-shot score and fine-tuning gain are closest to the benchmark means.
  • The two borderline pairs from the expert audit (partially answerable, caption partially needed) are shown in the new appendix "Expert audit of the benchmark" with the rater's verdicts.
  • The main text gains a section, "Does the benchmark require the image?"; its full table is in the new appendix "Image-dependence details", and per-level scores for all 12 systems are in a new appendix table.

W1 / Q1 Validation, and whether questions can be answered from the image alone. We agree the generator pilot (34 images, one annotator) was meant only to choose a generator, not to validate the corpus. We have added two forms of validation:

  1. Expert audit, image-only. An author with SEM expertise rated 50 rep300 pairs from 50 images (3.3% of pairs, 17% of images; 11–14 per level; all 15 cluster × centrality cells covered). Each pair was rated from the image alone before the caption was revealed, on the App. C rubric. Grounding good/average: 50/50 (good: 47/50); scientifically valid: 50/50; answerable from the image: 48/50 (partially: 2/50); hallucination: 0/50 (95% CI upper bound 7%); caption needed: 2/50, both only partially (one Identification, one Interpretation). The rater is an author; independent multi-rater validation remains future work, and we state this.
  2. Controlled image-dependence tests (image-level bootstrap 95% CIs; local Qwen3.5-4B judge). Zero-shot Qwen3.5-2B on rep300 scores 2.40 from the question alone, 2.59 from caption+summary without the image, and 2.94 with the image. The image beats caption+summary by +0.35 [+0.26, +0.43], and the result replicates on 1,000 random corpus images. When the fine-tuned model (B2-B) is shown an image from a different visual cluster, its score drops from 3.83 to 2.74 (paired drop 1.09 [1.01, 1.17]), versus 0.76 [0.68, 0.84] for the zero-shot model. Fine-tuned gains therefore depend on the correct image (difference +0.33 [+0.24, +0.43]). At the same time, about 0.57 of the 0.90 fine-tuning gain is image-independent (format and domain style), and we now report this decomposition.

On Identification, you are right, and the reason is our design. Captions and summaries guided question generation; answers are meant to come from the image. The image beats caption+summary for Observation, Detection (+0.65, the largest) and Interpretation. For Identification, caption+summary outperforms the image (−0.37): our grounding rule lets a technical name (phase, precipitate) be stated only when the caption corroborates it, so the feature is visible but its name is caption-confirmed. We now report this level separately and keep the Limitations statement that image-only answerability is a design goal, not a guarantee.

W2 The judge does not see the image. Agreed: it measures agreement with the reference, not image support. The tests above check image dependence directly, and the main text states this limitation.

Q3 Counts.

  • 47,981 vs 16,192: 47,981 panel images come from 16,192 source figures (multi-panel figures are split). The Reproducibility Statement wrongly used the figure count; corrected.
  • 1,497 is the full benchmark. One model's provider (GPT-5-mini) rejected one image (5 questions), leaving 1,492 answered by every system. For 20 of these, the judge did not return a usable score for every system, so 1,472 pairs are scored for all 12 systems; this is the common evaluation set. Computing each system's mean on these 1,472 pairs instead of on all its judged pairs changes no mean by more than 0.003 and leaves the ranking unchanged. All three numbers are now defined in the results section.

Q5 How regions are defined. Region-comparison questions are a small minority of rep300. Regions are described in natural spatial terms ("upper-left", "the panel with the 5 µm scale bar", "Zone I"), but you are right that the prompt did not require an anchor, and not every such question has one in the question text. App. A now states the anchor rule; the release will flag region questions without an anchor, and future generation runs will enforce it.

Reviewer_36WJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces SEM-VQA, a large-scale, open-ended visual question answering (VQA) dataset for scientific understanding of scanning electron microscopy (SEM) images. Using LLM-assisted annotation, the authors construct 224,026 question–answer–visual-evidence triplets from 47,981 SEM micrographs collected from the materials science literature. The questions are organized into four levels: Observation, Detection, Identification, and Interpretation. The generation pipeline incorporates explicit grounding and anti-hallucination constraints to encourage image-grounded answers and prevent unsupported scientific inferences.

The authors also construct a held-out benchmark comprising 300 SEM images and 1,497 QA pairs, selected through DINOv3 embedding-based clustering and stratified sampling to improve visual diversity. They evaluate open-weight and proprietary multimodal models using six automatic metrics and an LLM-as-a-judge protocol. Fine-tuning Qwen3.5-2B on SEM-VQA improves its judge score from 3.33 to 3.82, closing approximately 76% of the initial performance gap to the strongest proprietary baseline (GPT-5-mini, 3.98) and outperforming Gemini-3.1-Flash and Claude-Haiku-4.5.

Critical strengths

Large-scale, open-ended VQA datasets remain scarce in the SEM domain, where existing materials science benchmarks are relatively small and predominantly use multiple-choice questions. SEM-VQA addresses this gap by providing a substantial collection of domain-specific QA pairs, organized according to a four-level visual understanding taxonomy and accompanied by textual visual evidence. The authors also construct a visually diverse held-out benchmark and provide extensive comparisons across multimodal models. The consistent improvements across six fine-tuning configurations further demonstrate the practical value of domain-specific supervision. Overall, SEM-VQA could serve as a useful resource for training and evaluating vision-language models in materials science.

Critical weaknesses
  1. Limited evidence that the improvements reflect genuine visual understanding rather than adaptation to the annotation distribution.

My main concern is that the training and benchmark annotations are generated using the same LLM and prompting pipeline. This may introduce shared question patterns, answer styles, and annotation biases, making it difficult to determine whether the observed improvements arise from better image-grounded scientific understanding or from fitting the shared annotation distribution. This concern is further amplified by the evaluation protocol, as the LLM judge assesses agreement with the generated reference answers without directly examining the SEM images. Consequently, the reported gains do not fully establish improvements in genuine visual understanding.

  1. Limited and inconsistent cross-dataset generalization.

The authors include cross-dataset transfer experiments on four external materials science benchmarks in Appendix E.7, but the results do not fully address the above concern. Fine-tuning consistently improves performance on MaCBench and MatQnA, whereas all evaluated fine-tuned variants underperform the zero-shot baseline on MATRIX, with mixed results on CSMBench. Moreover, these experiments use a different LLM judge, making their absolute scores not directly comparable to the main results. These findings suggest that the benefits of SEM-specific adaptation are task-dependent. Therefore, the generalizability of the reported improvements beyond the original annotation distribution remains insufficiently demonstrated.

Questions for authors

Q1. Since the training and benchmark annotations are generated using the same LLM and prompting pipeline, how can the authors distinguish improvements in genuine image-grounded visual understanding from adaptation to the shared annotation distribution? Could the authors provide additional evidence, such as expert evaluation against the original SEM images or controlled tests of image dependence, to support their interpretation of the performance gains?

Q2. What is the added value of the four-level question taxonomy compared with directly generating SEM QA pairs without explicit level constraints? Is there experimental evidence that this taxonomy improves question diversity, coverage of microstructural features, or downstream model performance? An ablation comparing the proposed structured generation pipeline with a conventional QA generation approach would help clarify its contribution.

Q3. I noticed two inconsistencies in the manuscript. First, the main text reports 224,026 QA pairs from 47,981 SEM images, whereas the Reproducibility Statement reports the same number of QA pairs from only 16,192 images. Second, the abstract identifies GPT-5.4-mini as the strongest proprietary baseline (score: 3.98), whereas the main results tables report GPT-5-mini (score: 3.983). Could the authors clarify these discrepancies and confirm the dataset size and model version used in the experiments?

1 comment

We hope these new experiments and corrections resolve your concerns; if so, we would kindly ask you to consider updating your score. We are glad to clarify anything further. W1/Q1 Image dependence and the text-only judge. We agree that a text-only judge measures agreement with the reference, not image support. We therefore added three tests (image-level bootstrap 95% CIs, as the 1,497 items come from 300 images; one local judge, Qwen3.5-4B, same rubric):

  1. Input ablation (zero-shot Qwen3.5-2B, rep300). Question only: 2.40. Caption+summary, no image: 2.59. Image: 2.94. Image beats question-only by +0.53 [+0.46, +0.61] and beats caption+summary by +0.35 [+0.26, +0.43]. On 1,000 random corpus images the effects replicate (+0.44 [+0.40, +0.48]; +0.34 [+0.29, +0.39]). The benchmark therefore rewards looking at the image, not only producing the reference style.
  2. Fine-tuned model with a mismatched image (new). We gave the fine-tuned models an image from a different visual cluster (a bijective swap; no image keeps its own cluster), or no image at all. SFT B2-B falls from 3.83 with its own image to 2.74 with a mismatched image (paired drop 1.09 [1.01, 1.17]) and to 2.99 from the question alone. The zero-shot model drops only 0.76 [0.68, 0.84] under the same swap, so fine-tuning makes the model more dependent on the correct image, by +0.33 [+0.24, +0.43] (B1-A: +0.35 [+0.25, +0.44]). This image-specific gain is concentrated in Observation (+1.01) and Detection (+0.35); it is not significant for Identification or Interpretation. Your concern is also partly confirmed, and we report that too. Under this judge, about +0.57 of the +0.90 fine-tuning gain persists with a wrong image or no image: that part is image-independent adaptation (answer format, domain vocabulary, reference style). The paper now presents the gain decomposed this way, not as purely visual.
  3. Expert audit, image-only. A materials scientist (an author) rated 50 rep300 pairs (50 images, all 15 cluster × centrality cells) from the image alone, before seeing the caption, on the rubric of App. C. Grounding good/average: 50/50 (good: 47/50); valid: 50/50; answerable from the image: 48/50 (2 partially); hallucination: 0/50 (95% CI upper bound 7%). In the 2 borderline pairs the caption supplied an identity term (Ni, γ′), the same pattern as Identification below.

Captions and summaries guided question generation; the design goal is that answers come from the image. The per-level ablation shows the image beats caption+summary for Observation, Detection (+0.65, the largest) and Interpretation. Identification is the exception (−0.37), as our grounding rule predicts: a technical name (phase, precipitate) may be stated only when the caption corroborates it, so the feature is visible but its name is caption-confirmed. We now report this level separately.

W2 Cross-dataset transfer. We agree, and we have narrowed our claim. With all comparisons under the same judge, fine-tuning improves MaCBench (+0.12 to +0.25) and MatQnA (+0.19 to +0.34) for every variant, is mixed on CSMBench, and lowers MATRIX for every variant. We checked whether the MATRIX loss comes from its non-SEM items (EDS/XRD/TGA). It does not: on MATRIX's SEM-SE/BSE items alone, fine-tuned models also lose (best variant B2-B: −0.39 [−0.66, −0.12]). The revision states the scope plainly: SEM-VQA gives strong in-distribution gains and transfers to open-ended materials QA on two of four external benchmarks, but not generally. These use a different judge, so we compare only within that table.

Q2 Value of the four-level taxonomy. We did not run a no-taxonomy generation ablation, and we say so. The taxonomy serves two purposes that we can support with evidence:

  • Coverage control. 96.9% of images receive questions from at least three levels (47.5% from all four), so the questions go beyond description to detection and interpretation.
  • Diagnostics. The levels behave measurably differently: Detection is the most image-dependent, while Identification relies on caption-confirmed names (W1). Without the taxonomy this pattern would be invisible. We add a per-level table for all 12 systems (new appendix "Per-level scores and judge-bias check"); for example, zero-shot InternVL models vary by 0.42 across levels, versus 0.14 for our best fine-tuned model (B2-B).

Q3 — Inconsistencies. Both are errors in our text, and we have corrected them. (i) The 47,981 panel images come from 16,192 source figures (multi-panel figures are split; about 2.96 panels per figure). The Reproducibility Statement wrongly used the figure count. (ii) The evaluated model is GPT-5-mini (3.983). "GPT-5.4-mini" was our Azure deployment name and leaked into the abstract. We also fixed the abstract rounding to 3.82 and a broken section reference. (iii) The Gemini baseline is Gemini-3.1-Flash-Lite; the paper wrongly called it Gemini-3.1-Flash. All mentions are corrected.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.