- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SEM-VQA, a large VQA dataset for scientific understanding of SEM images. The dataset contains 224,026 question–answer–visual-evidence triplets derived from 47,981 SEM images collected from materials-science publications. Questions are generated using Gemini-2.5-Flash with image, caption, and summary inputs and organized into four levels: Observation, Detection, Identification, and Interpretation. The authors construct a held-out benchmark of 300 images and QA pairs using embedding-based clustering and evaluate multiple open-weight and proprietary multimodal language models using six automatic metrics and an LLM-based judge.
Critical strengths
S1. Large-scale dataset with structured question categories. The benchmark is much larger than existing materials-specific VQA benchmarks. The four-level taxonomy provides a useful framework for organizing questions about visual observation, feature detection, technical identification, and scientific interpretation. The inclusion of visual-evidence descriptions is also potentially valuable for developing interpretable scientific VQA systems.
Critical weaknesses
W1. My concern is about the reliability of the benchmark annotations. The entire corpus is generated using a single MLLM, and it is unclear how human evaluation is involved. Given the specialized nature of SEM interpretation, automatically generated answers may contain scientific inaccuracies.
W2. Limited size and validation of the held-out benchmark. Although the training corpus is large, the evaluation benchmark contains only 300 images. It would be better to report performance separately for the four question categories in Table 1.
W3. There are some inconsistencies. The main paper repeatedly reports 47,981 images, whereas the reproducibility statement on page 10 says 16,192 images. The abstract also refers to GPT-5.4-mini, while the main experimental tables identify GPT-5-mini.
Questions for authors
W2 and W3 in the above weakness section.
We thank the reviewer for the detailed review. We have uploaded a revised PDF; changes are summarised below.Kindly reconsider your score
W3 / Q2 / Q4 Figures and complete examples. We agree that this was the paper's main presentation gap, and we have fixed it.
W1 / Q1 Validation, and whether questions can be answered from the image alone. We agree the generator pilot (34 images, one annotator) was meant only to choose a generator, not to validate the corpus. We have added two forms of validation:
On Identification, you are right, and the reason is our design. Captions and summaries guided question generation; answers are meant to come from the image. The image beats caption+summary for Observation, Detection (+0.65, the largest) and Interpretation. For Identification, caption+summary outperforms the image (−0.37): our grounding rule lets a technical name (phase, precipitate) be stated only when the caption corroborates it, so the feature is visible but its name is caption-confirmed. We now report this level separately and keep the Limitations statement that image-only answerability is a design goal, not a guarantee.
W2 The judge does not see the image. Agreed: it measures agreement with the reference, not image support. The tests above check image dependence directly, and the main text states this limitation.
Q3 Counts.
Q5 How regions are defined. Region-comparison questions are a small minority of rep300. Regions are described in natural spatial terms ("upper-left", "the panel with the 5 µm scale bar", "Zone I"), but you are right that the prompt did not require an anchor, and not every such question has one in the question text. App. A now states the anchor rule; the release will flag region questions without an anchor, and future generation runs will enforce it.