- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 1(−2)
Read review
Paper summary
The paper addresses image-level attribute prediction from materials micrographs under heterogeneous supervision: a small set of images has instance masks, while a much larger set has only image-level attribute labels extracted from the literature. The proposed SGAP is a two-stage framework built on SAM3. Stage I trains the last two ViT blocks, the pixel decoder and a Cellpose-style structural head (foreground probability plus a center-pointing flow field) on the mask-annotated images. Stage II freezes this structural pathway and trains LoRA adapters and attribute heads on the mask-free images. Each image passes through the backbone twice: the LoRA-off pass produces a detached foreground probability map, which serves as a soft weight for pooling the features of the LoRA-on pass. The pooled features feed heads for porosity, grain size, aspect ratio, morphology and size uniformity, and an optional density-gated dual-expert head handles the bimodal porosity distribution.
The paper also contributes an open-sourced, self-built Si3N4 SEM dataset: about 2,970 images with LLM-extracted attribute labels from 1,639 papers, plus 41 expert-annotated images.
Validation uses two datasets. The first is a public stainless-steel grain dataset (480 images, with masks kept for 100 training images). The second is the ceramic dataset (41 / 2,377 / 593 images for Stage I / Stage II training / test, split by source publication). Baselines are segment-then-measure pipelines (SAM3, CPSAM, MatSAM) and zero-shot VLMs (GPT-5.5, Gemini-3.8-Flash). SGAP reports the lowest errors on most metrics. Ablations (Table 3, Fig. 5) examine the data-use strategy and the individual modules.
Critical strengths
-
A coherent combination of established techniques. Weighted average pooling, a dual forward pass with LoRA switched on and off, and mixed supervision are each well known. The paper nonetheless combines them into a reasonable pipeline that uses scarce masks for localization and abundant literature labels for regression, and the full system outperforms the reported segment-then-measure and VLM baselines on both datasets (Tables 1 and 2). For this reason I do not regard the methodological novelty as the main obstacle to acceptance.
-
An open-sourced, self-built dataset. Pairing literature micrographs with their reported attributes is a scalable source of supervision for materials science, and releasing such a dataset could benefit the community, provided its quality can be established (see Weakness 1 and 2).
Critical weaknesses
-
The reliability of the data, and therefore of the evaluation, is not established. The paper reports that a manual audit of the LLM-extracted labels gave 89.1% accuracy, but does not state how many records were audited, how they were sampled, or which criteria were used. The test set is drawn from the same self-built dataset, and no authoritative verification of the test labels is described; the validation set used for early stopping is not described at all. If the evaluation labels carry a comparable error rate (roughly 10%), the gains attributed to the proposed design are hard to interpret. The ablations suggest that most of the improvement comes from the additional attribute data together with backbone adaptation. In Fig. 5 / Table 5, porosity MAE drops from 15.93 (masked-only) to 7.00 once mask-free labels are used with a fine-tuned backbone, compared with 6.67 for the full model. The components the paper emphasizes contribute only about 2%: in Table 3, removing foreground-guided pooling changes porosity and grain-size MAE by +2.0% and +1.9%, and actually improves aspect-ratio MAE by 3.3%. With single-seed results and this level of label noise, such differences fall below what the data can resolve. The central claim, that the structure-guided design is responsible for the improvement, is therefore not convincing.
-
Insufficient data filtering and transparency. Images I randomly sampled from the anonymous repository include an image with an embedded inset sub-figure (9242be17_a.jpg) and a highly blurred image (77384a59_d.jpg). Neither is suitable for the quantitative measurements the paper targets. This suggests that the filtering step did not exclude such images and that the 89.1% figure may overestimate the actual quality of the image–label pairs. In addition, the paper does not appear to report which LLM was used to construct the dataset, which limits reproducibility and makes the extraction quality harder to assess. I believe at least an automated image-quality screening (for example, with a VLM) is needed so that such low-quality images do not enter training, let alone evaluation.
(Minor: In Figure 2, the “image” label overlaps the Stage II input micrograph and covers part of it, including the scale bar. Please fix the figure.)
Questions for authors
Clarifications (answers could change my assessment):
- For the 89.1% audit: how many records were audited, how were they sampled, and what criteria defined a correct record? Was the test set manually verified in full? If not, what is the estimated error rate of the test labels?
- Which data were used as the validation set for early stopping ("validation metrics", Sec. 4.1 and A.12)? Is it disjoint from the test set?
- Which LLM (name, version, decoding settings) was used for attribute extraction?
- What criteria does the "label-quality filtering" step (5,884 → 3,011 records) apply to the images? What fraction of the training and test images contain insets, multiple scale bars or heavy blur?
Suggested experiments:
- Apply an automated image-quality screening (e.g., VLM-based) and re-report the main results and the Table 3 ablations on the cleaned test subset.
- Report Table 3 over multiple random seeds (mean ± std) to show that the gains of the proposed components exceed run-to-run variance.
My current recommendation is weak reject. If the authors adequately address questions 1–4 and the data-quality concern, I will consider raising my score.
8 comments
Answer (Q2 & Q3)
Answer (Q2) Thank you for this question. All data used for model training and early stopping are disjoint from the test set. Within the training pool, we reserve approximately 20% of the images as a validation set for early stopping and checkpoint selection, while the remaining images are used for parameter optimization. Specifically, in Stage I, the 41 mask-annotated images are split into 33 training images and 8 validation images. In Stage II, the 2,377 images with attribute labels are split into 1,903 training images and 474 validation images. Early stopping and checkpoint selection in each stage use only the corresponding validation set; the test set is reserved for final evaluation. We will clarify these splits and the role of each subset in the revised manuscript.
Answer (Q3) We used GPT-5.4 for attribute extraction. The complete extraction prompt is provided in the appendix subsection “Prompts,” under “Data Extraction Prompt.” We will explicitly identify the extraction model in the dataset-construction description and provide the available inference settings.
Answer (W2+Q4)
Answer (W2+Q4) We would like to express our sincere gratitude for the time and professional effort you have dedicated to reviewing our manuscript. Image filtering criteria and quality screening. Our original image-screening pipeline identified candidate SEM images using figure-caption keywords, followed by VLM checks for the target grain microstructure and identifiable scale bars. Images containing insets or reduced sharpness were initially retained to reflect the conditions encountered when estimating attributes from literature-derived figures.
We conducted an additional VLM-based screening for embedded insets, multiple scale bars, and severe blur or pixelation. The screening-positive rates are:
| Screening criterion | Training records (N=2,377) | Test records (N=593) |
|---|---|---|
| Embedded insets | 131 (5.51%) | 40 (6.75%) |
| Multiple scale bars | 131 (5.51%) | 38 (6.41%) |
| Severe blur or pixelation | 473 (19.90%) | 133 (22.43%) |
| At least one of the three criteria | 627 (26.38%) | 179 (30.19%) |
We manually reviewed 182 test images: 179 images with at least one “yes” judgment among the three criteria, plus 3 additional images with no “yes” judgment but at least one unresolved judgment. Images confirmed to contain an embedded inset, multiple scale bars, or severe blur/pixelation were excluded from the strict evaluation subset. Of the 182 reviewed images, 126 were excluded and 56 were retained. Together with the unflagged test images, the resulting strict test subset contains 467 images, which we use for the supplementary re-evaluation. After cleaning the attribute annotations, 409 of these 467 records contain at least one valid attribute label and are available for label-based evaluation.
Answer (Q1)
Answer (Q1) Thank you for this question. We randomly sampled 55 image-associated material records and manually verified them against the source publications. A record was counted as correct only when the image was matched to the correct material sample and all recorded attributes were consistent with the source publication. Any error in image matching or attribute extraction caused the entire record to be counted as incorrect. Overall, 49 of the 55 records passed this verification, yielding a record-level accuracy of 89.1%.
Manual verification and test-label reliability. The test set was not manually verified in full. In our initial dataset-level audit, 49 of 55 randomly sampled records met the record-level correctness criterion (89.1%); this sample was not specific to the test set. Additionally, our expert team (consisting of two associate professors and a doctoral student) has now audited 55 test records against the original source documentation. A record was considered correct only if both the image–sample correspondence and all reported attributes matched the source. Of these 55 records, 51 met the criterion and four contained at least one mismatch, corresponding to an observed record-level error rate of 7.27% (4/55). In addition, all test images underwent VLM-based screening for embedded insets, multiple scale bars, severe blur or pixelation, and scale readability. This screening assessed image quality and did not verify the extracted attributes against the source papers. We will report the sampling procedure and audit details in the supplementary material of the revised manuscript.