acceptodds
Under review as a conference paper at ICLR 2027

From Localization to Quantification: Structure-Guided Attribute Prediction under Weak Supervision

Abstract

Quantitative microstructural analysis often relies on instance segmentation to estimate material attributes. However, dense instance annotations are costly to obtain, whereas image-level attribute labels are widely available in the literature and technical reports. This supervision gap hinders the effective use of abundant attribute labels while preserving essential structural information. To address this challenge, we propose Structure-Guided Attribute Prediction (SGAP), a two-stage framework that leverages scarce structural annotations alongside abundant image-level attribute labels. In the first stage, SGAP learns structural representations from a limited set of instance masks. In the second stage, a frozen structural pathway provides localization priors, while a LoRA-adapted attribute pathway extracts task-specific features for attribute prediction. SGAP uses segmentation-derived spatial priors to guide feature aggregation and predicts attributes through learned heads, without explicit per-instance geometric measurement. Experiments across two distinct scenarios and domains show that SGAP consistently outperforms both segment-then-measure approaches and direct prediction baselines, achieving state-of-the-art performance. These results support the use of segmentation-derived localization priors for attribute learning under heterogeneous supervision.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Microscopy Image Analysis, Microstructure Quantification, Material Attribute Regression, Structure-Guided Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EEAmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2(−1)
Soundness
1(−2)
Read review
Paper summary

The paper addresses image-level attribute prediction from materials micrographs under heterogeneous supervision: a small set of images has instance masks, while a much larger set has only image-level attribute labels extracted from the literature. The proposed SGAP is a two-stage framework built on SAM3. Stage I trains the last two ViT blocks, the pixel decoder and a Cellpose-style structural head (foreground probability plus a center-pointing flow field) on the mask-annotated images. Stage II freezes this structural pathway and trains LoRA adapters and attribute heads on the mask-free images. Each image passes through the backbone twice: the LoRA-off pass produces a detached foreground probability map, which serves as a soft weight for pooling the features of the LoRA-on pass. The pooled features feed heads for porosity, grain size, aspect ratio, morphology and size uniformity, and an optional density-gated dual-expert head handles the bimodal porosity distribution.

The paper also contributes an open-sourced, self-built Si3N4 SEM dataset: about 2,970 images with LLM-extracted attribute labels from 1,639 papers, plus 41 expert-annotated images.

Validation uses two datasets. The first is a public stainless-steel grain dataset (480 images, with masks kept for 100 training images). The second is the ceramic dataset (41 / 2,377 / 593 images for Stage I / Stage II training / test, split by source publication). Baselines are segment-then-measure pipelines (SAM3, CPSAM, MatSAM) and zero-shot VLMs (GPT-5.5, Gemini-3.8-Flash). SGAP reports the lowest errors on most metrics. Ablations (Table 3, Fig. 5) examine the data-use strategy and the individual modules.

Critical strengths
  • A coherent combination of established techniques. Weighted average pooling, a dual forward pass with LoRA switched on and off, and mixed supervision are each well known. The paper nonetheless combines them into a reasonable pipeline that uses scarce masks for localization and abundant literature labels for regression, and the full system outperforms the reported segment-then-measure and VLM baselines on both datasets (Tables 1 and 2). For this reason I do not regard the methodological novelty as the main obstacle to acceptance.

  • An open-sourced, self-built dataset. Pairing literature micrographs with their reported attributes is a scalable source of supervision for materials science, and releasing such a dataset could benefit the community, provided its quality can be established (see Weakness 1 and 2).

Critical weaknesses
  • The reliability of the data, and therefore of the evaluation, is not established. The paper reports that a manual audit of the LLM-extracted labels gave 89.1% accuracy, but does not state how many records were audited, how they were sampled, or which criteria were used. The test set is drawn from the same self-built dataset, and no authoritative verification of the test labels is described; the validation set used for early stopping is not described at all. If the evaluation labels carry a comparable error rate (roughly 10%), the gains attributed to the proposed design are hard to interpret. The ablations suggest that most of the improvement comes from the additional attribute data together with backbone adaptation. In Fig. 5 / Table 5, porosity MAE drops from 15.93 (masked-only) to 7.00 once mask-free labels are used with a fine-tuned backbone, compared with 6.67 for the full model. The components the paper emphasizes contribute only about 2%: in Table 3, removing foreground-guided pooling changes porosity and grain-size MAE by +2.0% and +1.9%, and actually improves aspect-ratio MAE by 3.3%. With single-seed results and this level of label noise, such differences fall below what the data can resolve. The central claim, that the structure-guided design is responsible for the improvement, is therefore not convincing.

  • Insufficient data filtering and transparency. Images I randomly sampled from the anonymous repository include an image with an embedded inset sub-figure (9242be17_a.jpg) and a highly blurred image (77384a59_d.jpg). Neither is suitable for the quantitative measurements the paper targets. This suggests that the filtering step did not exclude such images and that the 89.1% figure may overestimate the actual quality of the image–label pairs. In addition, the paper does not appear to report which LLM was used to construct the dataset, which limits reproducibility and makes the extraction quality harder to assess. I believe at least an automated image-quality screening (for example, with a VLM) is needed so that such low-quality images do not enter training, let alone evaluation.

(Minor: In Figure 2, the “image” label overlaps the Stage II input micrograph and covers part of it, including the scale bar. Please fix the figure.)

Questions for authors

Clarifications (answers could change my assessment):

  1. For the 89.1% audit: how many records were audited, how were they sampled, and what criteria defined a correct record? Was the test set manually verified in full? If not, what is the estimated error rate of the test labels?
  2. Which data were used as the validation set for early stopping ("validation metrics", Sec. 4.1 and A.12)? Is it disjoint from the test set?
  3. Which LLM (name, version, decoding settings) was used for attribute extraction?
  4. What criteria does the "label-quality filtering" step (5,884 → 3,011 records) apply to the images? What fraction of the training and test images contain insets, multiple scale bars or heavy blur?

Suggested experiments:

  1. Apply an automated image-quality screening (e.g., VLM-based) and re-report the main results and the Table 3 ablations on the cleaned test subset.
  2. Report Table 3 over multiple random seeds (mean ± std) to show that the gains of the proposed components exceed run-to-run variance.

My current recommendation is weak reject. If the authors adequately address questions 1–4 and the data-quality concern, I will consider raising my score.

8 comments
AuthorsOpenReview ↗

Answer (Q1)

Answer (Q1) Thank you for this question. We randomly sampled 55 image-associated material records and manually verified them against the source publications. A record was counted as correct only when the image was matched to the correct material sample and all recorded attributes were consistent with the source publication. Any error in image matching or attribute extraction caused the entire record to be counted as incorrect. Overall, 49 of the 55 records passed this verification, yielding a record-level accuracy of 89.1%.

Manual verification and test-label reliability. The test set was not manually verified in full. In our initial dataset-level audit, 49 of 55 randomly sampled records met the record-level correctness criterion (89.1%); this sample was not specific to the test set. Additionally, our expert team (consisting of two associate professors and a doctoral student) has now audited 55 test records against the original source documentation. A record was considered correct only if both the image–sample correspondence and all reported attributes matched the source. Of these 55 records, 51 met the criterion and four contained at least one mismatch, corresponding to an observed record-level error rate of 7.27% (4/55). In addition, all test images underwent VLM-based screening for embedded insets, multiple scale bars, severe blur or pixelation, and scale readability. This screening assessed image quality and did not verify the extracted attributes against the source papers. We will report the sampling procedure and audit details in the supplementary material of the revised manuscript.

AuthorsOpenReview ↗

Answer (Q2 & Q3)

Answer (Q2) Thank you for this question. All data used for model training and early stopping are disjoint from the test set. Within the training pool, we reserve approximately 20% of the images as a validation set for early stopping and checkpoint selection, while the remaining images are used for parameter optimization. Specifically, in Stage I, the 41 mask-annotated images are split into 33 training images and 8 validation images. In Stage II, the 2,377 images with attribute labels are split into 1,903 training images and 474 validation images. Early stopping and checkpoint selection in each stage use only the corresponding validation set; the test set is reserved for final evaluation. We will clarify these splits and the role of each subset in the revised manuscript.

Answer (Q3) We used GPT-5.4 for attribute extraction. The complete extraction prompt is provided in the appendix subsection “Prompts,” under “Data Extraction Prompt.” We will explicitly identify the extraction model in the dataset-construction description and provide the available inference settings.

AuthorsOpenReview ↗

Answer (W2+Q4)

Answer (W2+Q4) We would like to express our sincere gratitude for the time and professional effort you have dedicated to reviewing our manuscript. Image filtering criteria and quality screening. Our original image-screening pipeline identified candidate SEM images using figure-caption keywords, followed by VLM checks for the target grain microstructure and identifiable scale bars. Images containing insets or reduced sharpness were initially retained to reflect the conditions encountered when estimating attributes from literature-derived figures.

We conducted an additional VLM-based screening for embedded insets, multiple scale bars, and severe blur or pixelation. The screening-positive rates are:

Screening criterionTraining records (N=2,377)Test records (N=593)
Embedded insets131 (5.51%)40 (6.75%)
Multiple scale bars131 (5.51%)38 (6.41%)
Severe blur or pixelation473 (19.90%)133 (22.43%)
At least one of the three criteria627 (26.38%)179 (30.19%)

We manually reviewed 182 test images: 179 images with at least one “yes” judgment among the three criteria, plus 3 additional images with no “yes” judgment but at least one unresolved judgment. Images confirmed to contain an embedded inset, multiple scale bars, or severe blur/pixelation were excluded from the strict evaluation subset. Of the 182 reviewed images, 126 were excluded and 56 were retained. Together with the unflagged test images, the resulting strict test subset contains 467 images, which we use for the supplementary re-evaluation. After cleaning the attribute annotations, 409 of these 467 records contain at least one valid attribute label and are available for label-based evaluation.

Reviewer_LkP5OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Tha paper presents a framework for microstructural analysis of microscopy images composed by two stages; in the first a segmentation module based on SAM learns structural representations from a limited set of instance masks; in the second, localization priors together with a LoRA-adapted attribute pathway is exploited for attribute prediction. Authors demonstrate the capabilities of the framework in the context of various material science benchmarks, and compare with baselines used for attribute prediction.

Critical strengths
  1. Paper polished and well presented.
  2. System carefully designed, even if the main components do not appear to contain strong novelty for what concerns the representation learning.
  3. Results appear convincing, and showcase the potential of the framework for material science
Critical weaknesses
  1. Lack of generality: even if the title claims for generality, the framework appears to be customized for material science, and the contributions in the sense of learning representation appear marginal. I would have expected efforts to showcase validity even for other domains using microscopy images, like ultrastructural analysis for neuroscience, or histopathology. In this condition, the manuscript appears more adequate for material science venues.
  2. Lack of novelty: despite the system is carefully designed, the formulation of the problem and the components are well known, and the manuscript does not look to provide enough contributions for the 'learning' community.
Questions for authors

Showcase the capability of the proposed framework in different domains, apart of material science.

2 comments
AuthorsOpenReview ↗

Answer for question

Thank you for raising the questions of generality and novelty. We respectfully submit that the contribution is not limited to a new representation-learning module. SGAP addresses an important learning setting in which dense pixel-level annotations are scarce, while image-level attribute labels are substantially more available. Its central contribution is a two-stage framework that uses the scarce masks to learn structural localization and then uses abundant image-level labels to adapt attribute prediction while retaining that structural prior. This allows the larger weakly labeled collection to contribute supervision without requiring masks for every image. We agree that the individual building blocks are established techniques. Our contribution is their organization around this supervision imbalance and the empirical study of how the complementary annotations can be used together. The experiments cover ceramic and stainless-steel microstructures, with different material systems and prediction targets. Our experiments on stainless-steel and ceramic microstructures examine the framework across different material systems and microstructural appearances, providing evidence of its applicability within materials microscopy. We have not yet evaluated SGAP outside materials science. Accordingly, the current results establish applicability across the evaluated material systems, rather than cross-domain generalization to neuroscience or histopathology. We will revise relevant statements to make this empirical scope explicit. The two-stage learning formulation can, in principle, be applied to other domains with similar supervision. We regard evaluation on biological and medical microscopy as an important next step. The present work establishes a concrete framework and empirical evidence for learning from complementary annotation sources in materials microscopy, which remains a meaningful learning problem even within this validated scope. We ask that the work’s contribution be assessed in terms of this annotation-efficient learning problem and the evidence provided, rather than whether each component introduces a new representation-learning operation.

Reviewer_LkP5OpenReview ↗

Unchanged mind

Considering the issues raised by other reviewers, and the fact that my concerns do not appear to be adequately addressed, I do not feel to change my rating for this manuscript

Reviewer_eMEVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Structure-Guided Attribute Prediction (SGAP), where segmentation-derived information is used as a spatial prior for direct attribute prediction. Specifically, a SAM3-based dense visual encoder and pixel decoder are combined with a Cellpose-style structural head to learn foreground and center-directed flow representations from the limited mask-annotated subset. Then, a LoRA-adapted attribute pathway extracts task-specific features for attribute prediction. The method is evaluated on a public stainless-steel grain-boundary dataset and a literature-derived ceramic SEM dataset. The reported results are generally strong.

Critical strengths
  1. The proposed setting of Nm≪NaN_m \ll N_a is realistic for scientific imaging and gives the work practical relevance beyond simply improving segmentation accuracy.
  2. The proposed pipeline is sound. The weak image-level annotation helps identify structural-level understanding.
  3. The authors make a meaningful effort to validate the automatically extracted labels.
  4. The numerical gains on several attributes are substantial rather than marginal.
Critical weaknesses
  1. The methodological novelty appears moderate. Most SGAP components are established: SAM/SAM3, Cellpose-style structural supervision, LoRA adaptation, stop-gradient separation, spatially weighted pooling, and mixture-of-experts regression. The main novelty seems to lie in their integration into this heterogeneous-supervision setting rather than in a fundamentally new learning mechanism. Also, applying the method to materials-property prediction seems new. But inherently, it is closely related to weakly supervised segmentation/localization, and segmentation-guided classification/regression, and multi-task prediction, which are more fundamental and general computer vision tasks and have been widely studied. The authors should position the paper toward applications rather than methodology.

  2. Why and how stage-I works? The SGAP generates a structural probability map, which is used as a soft spatial weight to average the LoRA-adapted features. It is not clear whether SGAP discovers attribute-specific spatial evidence or whether the authors prescribe where each attribute should look. For example, for a new material attribute, it is unclear how the appropriate structural prior would be determined without expert knowledge. Will this model fail? This does not make the method incorrect, but it makes the conceptual story less clean and complicates the interpretation that stage I learns a purely structural prior.

  3. The optional dual-expert component is somewhat task-specific and insufficiently justified. A threshold is used to separate low- and high-value regimes. And on the stainless-steel SPAF task, the dual-expert variant actually performs slightly worse than ordinary SGAP on several metrics. What principle can be used to determine which version of SGAP should be used?

  4. Only segment-then-measure pipelines and zero-shot VLMs are chosen as baselines. Intuitively, a fully matched supervised model should be considered.

Questions for authors

Q1. Novelty. Could the authors more carefully distinguish SGAP from related fields mentioned in Weakness-1 above?

Q2. What measurable benefit is provided by the flow prediction? Please provide an ablation removing LflowL_{\text{flow}} or replacing the Cellpose-style structural head with a simple binary foreground segmentation head.

Q3. Fully matched direct-prediction baseline mentioned in Weakness-4 above.

Q4. Is localization quality correlated with downstream attribute error? Demonstrating such a relationship would substantially strengthen the claim that SGAP provides meaningful and interpretable structural evidence rather than simply an additional regularizer.

Q5. How was the 10% threshold for the dual-expert model selected? Was it fixed before evaluation or chosen based on validation data? Please provide threshold sensitivity experiments.

Q6. The paper reports a manually audited extraction accuracy of 89.1%. How is it obtained?

7 comments
AuthorsOpenReview ↗

Answer (W1+Q1)

Answer (W1+Q1) Thank you for this comment. We agree that the contribution of SGAP is best described as an application-oriented learning framework rather than a new segmentation architecture. SGAP jointly exploits two complementary sources of supervision: a small set of images with instance-mask annotations and a much larger set of images with low-cost image-level attribute labels. The two-stage design prevents either source from being discarded: Stage I learns a structural representation from the mask-annotated subset, while Stage II uses the learned structural prior together with the abundant attribute labels to train attribute predictors without requiring masks for every image. This setting is important because using only mask-annotated images would discard most available attribute supervision, whereas using only image-level labels would lose explicit structural guidance. The contribution is therefore the supervision-efficient training and information-sharing strategy for this heterogeneous annotation setting, while SAM3, Cellpose-style flow supervision, and LoRA are established building blocks.

AuthorsOpenReview ↗

Answer (W4+Q3)

Answer (W4+Q3) Thank you for your comment. Our w/o Foreground-Guided Pathway ablation provides this matched direct-prediction comparison. It starts from the same Stage-I mask-trained checkpoint and uses the same attribute data, data split, random seed, optimization protocol, and early-stopping rule as SGAP. It replaces structural-prior-guided pooling with global average pooling and removes the separate structural pathway. Its results are reported in Table 3: porosity MAE/R2R^2 = 6.800/0.704, grain-size MAE/R2R^2 = 1.099/0.150, aspect-ratio MAE/R2R^2 = 1.871/0.581, morphology bAcc/F1 = 68.52/0.688, and size-uniformity bAcc/F1 = 61.09/0.602. Compared with the full SGAP model, this baseline is slightly better for aspect-ratio prediction but weaker on the other tasks. We will explicitly identify it as a fully matched image-level direct-prediction baseline in the revised manuscript.

AuthorsOpenReview ↗

Answer (W2+Q4)

Answer (W2+Q4) Thank you for this comment. We examined the relationship between localization quality and downstream attribute error using five-fold out-of-fold predictions. Higher grain Dice was associated with lower porosity absolute error (Spearman’s ρ = −0.679, n = 25 images). This negative association is favorable: images with more accurate grain localization tended to have more accurate porosity estimates. Grain Dice showed weak negative correlations with grain-size and aspect-ratio absolute errors (ρ = −0.133, n = 50; and ρ = −0.201, n = 50, respectively). Pore Dice also showed a weak negative correlation with porosity absolute error (ρ = −0.251, n = 25).

The weaker associations for grain size and aspect ratio may partly reflect the smaller number of corresponding attribute labels available for Stage II training (964 and 658, respectively), compared with porosity (2,599). In addition, pore regions occupy a relatively small portion of an image, so minor boundary differences can substantially affect pore Dice without producing a proportional change in image-level porosity error.These factors may contribute to the attribute-dependent patterns observed in our analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.