acceptodds
Under review as a conference paper at ICLR 2027

MatMicroBench: Towards Realistic Agentic Materials Microstructure Characterization

Abstract

Real-world microstructural characterization requires a continuous analysis process that observes relevant scientific evidence, models analysis rules as executable procedures, measures quantitative characteristics, and reasons from the resulting evidence and measurements to scientific conclusions. Existing benchmarks, however, largely focus on knowledge-centric or single-step tasks and provide limited coverage of such end-to-end analysis in realistic experimental settings. We introduce MatMicroBench, a benchmark toward realistic agentic materials microstructural characterization, where agents operate over experimentally acquired data and complete long-horizon scientific analysis with multiple interdependent outputs. Built from controlled laboratory experiments, MatMicroBench contains 64 instances based on SEM and EBSD data and spans four complementary tasks: static defect-state analysis, temporal defect-evolution analysis, inter-grain structure analysis, and intra-grain orientation-variation analysis. Each task requires agents to jointly produce spatial evidence, executable code, numerical results, and scientific conclusions, enabling capability-resolved evaluation of Observe, Model, Measure, and Reason. We evaluate 15 frontier models under a fixed execution harness and further compare six representative models across multiple agent harnesses. Under the fixed harness, even the strongest models achieve only around 53 points overall, and different models lead on different tasks, indicating substantial room for improving end-to-end microstructural analysis. Capability analysis further reveals task-specific bottlenecks, while strong code execution does not necessarily translate into accurate scientific outputs. These findings expose distinct failure modes across the characterization process and establish MatMicroBench as a testbed for developing more reliable agents for real-world materials analysis.

Primary area:
datasets and benchmarks
Keywords:
Materials Microstructural Characterization, Agent Benchmark, Long-Horizon Scientific Workflows

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tnGGOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MatMicroBench, an agentic platform for microstructural characterization of materials, where agents operate on experimental data to carry out well defined data processing tasks with the final aim of providing valuable scientific conclusions. MatMicroBench considers for four basic capabilities for microstructural analysis, namely, Observe, Model, Measure, and Reason. A comprehensive experimental set-up is presented where 15 frontier models are assessed. The analysis pinpoints task-specific bottlenecks as well as variable harness effects.

Critical strengths

The key contribution of this paper is the proposal of MatMicroBench, an agentic AI platform for end-to-end microstructural characterization of materials. The very comprehensive experimental results represents another benefit of this work. The paper is written in a clear and easy to understand style.

Critical weaknesses

The major weakness is the reduced technical content: no novel AI/ML tools are developed or proposed in this study. The whole paper seems like a case study---the whole work resumes to running a comprehensive set of experiments. Almost the whole body of the paper consists of describing the experiments and interpreting the results.

An issue that is not clear: the authors state on page 2: "the value of MATMICROBENCH in identifying capability bottlenecks and guiding the development of more reliable agents for microstructural characterization". The second claim "guiding the development of more reliable agents for microstructural characterization" is not at all justified or demonstrated in the paper. What is the strategy to develop "more reliable agents"? What triggered such a statement? There are many ways to " identifying capability bottlenecks". It is not clear why these two aspects are emphasized by the authors. No comparison or referencing is mentioned vis-a-vis other methods.

Questions for authors

It would be nice to see a strategy to optimize the agentic AI models considered in this work.

2 comments
AuthorsOpenReview ↗

Official Response by Authors to Reviewer tnGG

Q: Beyond benchmarking, how can the identified capability bottlenecks guide concrete strategies for developing more reliable scientific agents?

A: We thank the reviewer for this valuable suggestion. We agree that translating benchmark findings into actionable agent optimization strategies is important. Although MatMicroBench does not propose a new optimization algorithm, its technical contribution extends beyond a conventional benchmark or experimental case study. By decomposing real-world scientific workflows into four capabilities (Observe, Measure, Model, and Reason) and evaluating intermediate artifacts alongside final scientific conclusions, MatMicroBench enables systematic diagnosis of where and how scientific agents fail, rather than merely reporting overall performance. This distinction is particularly important for scientific tasks, where successful tool execution does not necessarily guarantee scientifically valid results.

Our findings provide concrete evidence for capability-specific optimization. As shown in Section 4.3, the dominant bottlenecks differ across tasks: Observe/Measure for SEM-S, Reason for SEM-T, Measure for EBSD-S, and Observe for EBSD-O. Notably, the relatively weak correlation between Model and Reason in SEM-T (ρ = 0.46) indicates that successful modeling does not necessarily translate into reliable scientific interpretation. Together with the task-dependent harness effects observed in Section 4.4, these results suggest that simply improving foundation models or general-purpose tool use may be insufficient. Instead, different scientific failure modes call for different optimization strategies.

Specifically, MatMicroBench can inform two promising directions:

1) Domain-specific skill augmentation. Capability bottlenecks can guide the design of reusable scientific skills that integrate domain knowledge, specialized tools, and standardized analysis procedures. For example, EBSD-O motivates crystallography-aware skills for interpreting orientation maps and identifying target grains, whereas EBSD-S calls for more reliable grain measurement and verification procedures. Similarly, SEM-S motivates defect-aware segmentation and quantitative characterization skills. Such targeted augmentation can address domain-specific limitations without requiring improvements to every aspect of the underlying model.

2) Trajectory-level optimization and self-improvement. Beyond final-task rewards, capability scores and intermediate artifacts can potentially serve as structured feedback for agentic reinforcement learning or recursive self-improvement (RSI). For example, EBSD-O could provide feedback on grain identification and orientation analysis, while SEM-T could emphasize damage-evolution reasoning rather than code execution alone. This would allow optimization to target specific failure steps, support corrective tool use, and improve the reliability of multi-step scientific reasoning.

We emphasize that these optimization strategies are prospective directions, not experimentally validated claims of the present paper. The contribution of MatMicroBench is to establish an experiment-grounded diagnostic and evaluation framework that identifies where optimization is needed and provides measurable criteria for assessing whether targeted improvements are effective. We will clarify this distinction in the revised manuscript and pursue skill-based and trajectory-level agent optimization in future work. We also hope MatMicroBench will provide a foundation for other researchers developing more reliable scientific agents.

I would like to thank the authors for their efforts to answer my comments. The paper's contributions have not increased substantially to justify my increase of score. This the reason for keep my score intact.

Reviewer_P7s3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MatMicroBench, a benchmark for evaluating AI agents on end-to-end materials microstructure characterization. It comprises 64 instances derived from controlled SEM and EBSD experiments, organized into four tasks covering static defect analysis, temporal defect evolution, inter-grain structure, and intra-grain orientation variation. The benchmark evaluates four capabilities: Observe, Model, Measure, and Reason, through spatial evidence, executable code, numerical measurements, and scientific conclusions, respectively. The authors evaluate 15 frontier models under a common execution harness and six models under alternative harnesses. The best model reaches an overall score of 53.38 under the common harness. The results reveal considerable variation across tasks and capabilities, particularly the gap between successful execution of analysis code and accurate scientific measurements or interpretations.

Critical strengths

S1. Grounded experimental study with evaluation on complete characterization workflows. Newly acquired SEM and EBSD data is used for testing. The tests span over four tasks with diversified aspects on microstructural analysis.

S2. The valuation provides some diagnostic insights. The evaluation enables analysis on identifying where agents may fail. Harness comparison provides practical evidence on model selection and execution environment.

Critical weaknesses

W1. The evaluation scope is in general limited.

The benchmark contain only 64 instances and 10 each for EBSD-S and EBSD-O. How well the observed model rankings generalize across materials, acquisition conditions and experimental specimens lacks more sufficient experimental justification. The evaluation relies on weights and an LLM judge that are calibrated over only four instances and limited number of model outputs. Broader judge validation, weight sensitivity and ranking uncertainty needs to be strengthened.

W2. The comparison with specialized microstructure characterization methods are limited.

The evaluation focuses on general-purpose frontier models and agent harnesses. It is less clear how their performance compares with existing specialized approaches. Some examples include:

  • Taufique et al., “Generalizable Image Segmentation for Microstructure Characterization Through Integrated SEM and EBSD Analysis” (npj Computational Materials, 2025), demonstrate SEM-based grain segmentation validated against EBSD measurements.

  • Mueez and Vyas, “Bridging Foundation Models and ASTM Metallurgical Standards for Automated Grain Size Estimation from Microscopy Images” (CVPR Workshops, 2026), evaluate a segmentation-to-quantification pipeline against U-Net, MatSAM, and a vision-language model. These are not direct competitors but are useful references for Observe and Measure, hence deserve a comparison.

Questions for authors

Q1 (W1, Section 3.3, p. 5; Appendix D.3, p. 21). Given the relatively small number of experimental specimens, could the authors clarify the diversity of materials and imaging conditions represented? How stable are model rankings under specimen-level resampling, repeated agent runs, or alternative task weighting?

Q2 (W1, Section 3.4, p. 6; Appendices C.3 and D.4, pp. 17–18 and 22). How sensitive are overall scores and model rankings to expert-defined weights and LLM-judge variability? Could the authors provide rank correlation or agreement analyses across alternative weights and human judgments?

Q3 (W2, Section 4.1, p. 6). Could the authors compare representative agents against a specialized image-analysis or segmentation-and-measurement pipeline on at least one SEM or EBSD task?

3 comments
AuthorsOpenReview ↗

Official Response by Authors to Reviewer P7s3 (1/3)

We thank the reviewer for recognizing MatMicroBench's experimental grounding and diagnostic value. We provide additional analyses addressing dataset diversity, ranking robustness, and evaluation reliability.

Part 1: Dataset Scope and Ranking Stability

Q1: How diverse is the experimental dataset, and are model rankings stable under specimen-level resampling, repeated runs, and alternative task weighting?

A: We appreciate the reviewer's suggestions and conducted three complementary experiments. All consistently preserve the leading model group.

Dataset diversity. MatMicroBench comprises 64 real experimental SEM/EBSD instances across four tasks: SEM-S, SEM-T, EBSD-S, and EBSD-O. The data include surface and cross-sectional observations at different magnifications, covering diverse microstructural features and analysis objectives.

1) Specimen-level resampling: The leading group remains stable.

We performed 10,000 specimen-level bootstrap resamples across all 15 models, preserving related SEM-T instances and paired EBSD specimens. The three leading models remain in the Top-3 in nearly all resamples, demonstrating robustness to specimen composition.

ModelOverall95% Bootstrap CITop-3 frequency
Claude Opus 553.38[51.44, 55.26]100.00%
GPT-5.6 Sol53.33[51.00, 55.56]99.66%
GPT-5.552.95[51.52, 54.44]99.96%
Gemini 3.7 Flash49.04[47.11, 50.84]0.01%

2) Repeated executions: The same leading group is maintained across runs.

We conducted three runs for four representative models on a fixed 32-instance subset, stratified across the four tasks (10 SEM-S, 12 SEM-T, 5 EBSD-S, and 5 EBSD-O) with specimen-group constraints. The observed standard deviations range from 0.73 to 1.13 points, and the same Top-3 group is preserved across runs. This supports leading-group stability under execution randomness within the evaluated subset.

ModelRun 1Run 2Run 3Mean ± SD
Claude Opus 552.0750.2251.7451.34 ± 0.99
GPT-5.6 Sol52.2550.3551.4251.34 ± 0.95
GPT-5.552.6253.4454.0853.38 ± 0.73
Gemini 3.7 Flash48.8148.0946.5947.83 ± 1.13

3) Alternative task weighting: Rankings remain highly consistent under different aggregation schemes.

We compared the original equal-task weighting with instance-count weighting and 1,000 random task-weight perturbations (±20%) across all 15 models. The minimum Spearman correlation across random perturbations was 0.9857, and the Top-3 model set remained unchanged.

ConfigurationSpearman ρ ↑Kendall τ-b ↑Top-3 overlap
Instance-count weighting0.99290.9619100%
Task weights ±20% (mean)0.99670.9845100%

Together, these experiments support ranking stability across specimen sampling, repeated executions, and task aggregation.

AuthorsOpenReview ↗

Official Response by Authors to Reviewer P7s3 (2/3)

Part 2: Reliability of Evaluation Scores

Q2: How were capability weights and numerical tolerances determined, and are they scientifically justified?

A: Capability weights reflect scientific requirements, as agreed upon by at least two materials experts. Numerical tolerances follow task-specific procedures:

  • SEM-S: Five-fold validation P90 errors on an independent development set.
  • SEM-T: Maximum of instance-specific Last-frame Persistence MAE and one-pixel area fraction.
  • EBSD-S/O: Maximum of reconstruction-error P75 and expert-defined floors.

These procedures provide task-specific reference scales. Complete configurations, per-instance tolerances, and scoring implementations will be released together with the code and dataset.

Q3: Could alternative capability weights substantially change the reported model rankings?

A: We conducted sensitivity analyses across all 15 models and 64 instances, comparing equal capability weights with 1,000 random weight perturbations (±20%). Model rankings remain highly correlated with the original results, and the Top-3 model set is preserved across all tested configurations. These findings demonstrate the robustness of the main comparative conclusions to capability-weight choices, although absolute scores may vary.

ConfigurationSpearman ρ ↑Kendall τ-b ↑Top-3 overlap
Equal capability weights0.97500.9048100%
Capability weights ±20% (mean)0.99780.9886100%

Q4: Can LLM-based Reason evaluation be trusted, particularly when the judge and evaluated agents may share model families?

A: We appreciate the reviewer's concern about LLM-judge variability. We strengthened Reason evaluation through cross-family judge comparisons and expanded human validation, both of which support the consistency of the main findings.

Cross-family validation. Re-evaluating all 15 agents with Claude Sonnet 5 and Gemini 3.1 Flash Lite yielded Spearman correlations of 0.9857 and 0.9714 against GPT-5.4, respectively. Both preserved the same Top-3 models.

Alternative judge vs. GPT-5.4Spearman ρ ↑Kendall τ-b ↑
Claude Sonnet 50.98570.9429
Gemini 3.1 Flash Lite0.97140.8857

Human validation. We expanded validation from 20 to 100 outputs, independently assessed by two materials experts across all four tasks. Spearman and ICC assess overall Reason scores and rubric-item agreement measures criterion-level consistency. Human–LLM agreement is comparable to inter-expert agreement, further supporting Reason evaluation reliability.

ComparisonSpearman ρ ↑ICC(A,1) ↑Rubric-item agreement ↑
Expert 1 vs. GPT-5.40.8670.85386.05%
Expert 2 vs. GPT-5.40.9330.93890.53%
Expert 1 vs. Expert 20.9130.91884.68%

AuthorsOpenReview ↗

Official Response by Authors to Reviewer P7s3 (3/3)

Part 3: Comparison with Specialized Microstructure Analysis Methods

Q5: How do agents compare with specialized segmentation methods, and is the comparison fair for semantic microstructure analysis?

A: We thank the reviewer for suggesting specialized methods. We conducted comparisons on all 20 SEM-S instances and found that leading agents outperform SAM 3 in semantic defect segmentation and achieve competitive structural segmentation results.

Baseline selection. We examined both recommended studies. Taufique et al. [1] focus on SEM-based grain boundary segmentation, while Mueez and Vyas [2] address grain segmentation and ASTM grain-size estimation. These differ from SEM-S, which requires semantic identification and quantification of pores, cracks, and semi-molten particles. We therefore selected Otsu [3], a classical thresholding method, and MatSAM [4], a materials-adapted segmentation model, which are used as baselines in the second study. Since neither directly supports semantic target prompting, we additionally included SAM 3 [5], which supports natural-language-guided segmentation.

Evaluation protocol. All methods use the same 20 instances and ground truth. SAM 3 receives separate prompts for three defect categories. MatSAM uses automatic segmentation, with all retained masks combined without manual or ground-truth-based selection. Semantic and class-agnostic results are reported separately.

1) Semantic defect segmentation: Leading agents outperform the tested SAM 3 baseline. Claude Opus 5 achieves 0.4052 mDice versus 0.0503 for SAM 3, highlighting the difficulty of text-guided segmentation for specialized microstructural categories.

MethodPore Dice ↑Crack Dice ↑Particle Dice ↑mDice ↑mIoU ↑Area MAE ↓
SAM 30.00000.01480.13600.05030.028924.532
GPT-5.50.59330.12770.16310.29470.210010.110
GPT-5.6 Sol0.66080.27750.22280.38710.28148.315
Claude Opus 50.63970.32370.25220.40520.29247.840

2) Class-agnostic structural segmentation: Leading agents achieve higher union Dice, while specialized methods retain advantages in precision or recall. Claude Opus 5 and GPT-5.6 Sol achieve higher union Dice than the tested Otsu and MatSAM baselines. However, Otsu achieves the highest precision, while MatSAM achieves the highest recall, reflecting different segmentation characteristics.

MethodUnion Dice ↑Union IoU ↑Precision ↑Recall ↑Area MAE ↓
Otsu0.51860.37430.95750.383320.810
MatSAM0.41620.28990.32730.811749.786
SAM 30.31270.20280.53020.569851.915
GPT-5.50.52710.37140.63350.525113.825
GPT-5.6 Sol0.60330.45980.72540.601414.614
Claude Opus 50.60720.46290.71840.626115.456

Area MAE is reported in percentage points (pp). Class-agnostic evaluation uses the pixel-wise union of defect masks.

3) Qualitative comparison. We further provide five representative SEM-S cases to visualize the segmentation performance of different methods (https://postimg.cc/kV3RvScZ). The results show that leading agents produce defect segmentation results closer to ground truth, while SAM 3 exhibits substantial errors and Otsu/MatSAM lack semantic category identification.

Together, these comparisons highlight agents' advantages in semantic defect identification and support evaluating scientific workflows that integrate identification, measurement, and interpretation.

[1] Taufique, Mohammad Fuad Nur, et al. "Generalizable image segmentation for microstructure characterization through integrated SEM and EBSD analysis." npj Computational Materials 11.1 (2025): 323.

[2] Mueez, Abdul, and Shruti Vyas. "Bridging Foundation Models and ASTM Metallurgical Standards for Automated Grain Size Estimation from Microscopy Images." arXiv preprint arXiv:2604.18957 (2026).

[3] Otsu, Nobuyuki. "A threshold selection method from gray-level histograms." IEEE Transactions on Systems, Man, and Cybernetics 9.1 (1979): 62–66.

[4] Li, Changtai, et al. "MatSAM: efficient extraction of microstructures of materials via visual large model." arXiv preprint arXiv:2401.05638 (2024).

[5] Carion, Nicolas, et al. "SAM 3: Segment Anything with Concepts." International Conference on Learning Representations (2026).

Reviewer_2Y7pOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MatMicroBench, an experiment-grounded benchmark for agentic microstructural characterization. It defines four capabilities—Observe, Model, Measure, and Reason—and evaluates them through four SEM/EBSD tasks. Each task requires spatial evidence, executable code, numerical measurements, and scientific conclusions. The benchmark contains 64 laboratory-derived instances. Fifteen models are evaluated under the Terminus-2 harness, and six models are compared across three harnesses. The best reported Overall score is 53.38.

Critical strengths
  1. The benchmark uses experimentally acquired SEM and EBSD data, including temporal SEM sequences and co-registered EBSD maps. This makes the tasks more realistic than knowledge QA or isolated image-understanding benchmarks.
  2. The artifact-level evaluation is well motivated. Requiring spatial evidence, executable procedures, numerical results, and scientific conclusions enables more detailed diagnosis of agent failures than evaluating only a final answer.
Critical weaknesses
  1. The evaluation score is not sufficiently calibrated. The benchmark uses expert-defined capability weights and numerical tolerances, but the complete configuration and sensitivity to these choices are not clearly presented in the manuscript. Model scores for the executable-code capability are often close to saturation, reducing discrimination. Scientific conclusions are judged by GPT-5.4, which is also one of the evaluated models, while the human validation study covers only 20 outputs. The judge’s access to raw images, masks, numerical results, and code is also unclear. These issues make the absolute scores and Reason rankings difficult to interpret. The authors should release the complete scoring configuration, report raw component metrics, validate the judge with independent evaluators, and include stronger calibration baselines.
  2. The SEM-T task and the benchmark scope need stronger qualification. SEM-T evaluates future-image prediction using FSIM and 1−1-GMSD against one realized future frame. These generic image-similarity metrics may not measure defect localization, newly formed damage, or morphology evolution accurately. The benchmark also contains a relatively small and correlated specimen set: 24 SEM-T instances come from six specimens, while EBSD-S and EBSD-O reuse the same ten specimens. The results are valuable as a controlled benchmark study, but broader claims about generalization across materials, instruments, or experimental conditions should be narrowed or supported by additional splits.
Questions for authors
  1. What are the exact capability weights, numerical tolerances, and hidden tests used to evaluate executable code?
  2. What inputs does the GPT-5.4 judge receive, and how is potential evaluator/model-family bias controlled?
  3. Does SEM-T include persistence or classical forecasting baselines, defect-aware metrics, and uncertainty analysis?
  4. Are the 64 instances grouped or split at the specimen level when estimating performance?
  5. Were results obtained from one execution per instance or repeated executions? How stable are close model rankings under repeated runs?
2 comments
AuthorsOpenReview ↗

Official Response by Authors to Reviewer 2Y7p (1/2)

We thank the reviewer for recognizing our experiment-grounded tasks and artifact-level evaluation. We provide additional analyses of scoring reliability and robustness.

Part 1: Reliability of Evaluation Scores

Q1: Are capability weights and numerical tolerances sufficiently justified, and could these scoring choices affect model rankings?

A: We clarify the scoring configurations and conduct sensitivity analyses showing that the Top-3 model set remains unchanged under the tested configurations. Complete configurations, per-instance tolerances, and scoring implementations will be released together with the code and dataset to support reproducibility.

Scoring configuration. Capability weights reflect scientific requirements, as agreed upon by at least two materials experts.

TaskObserveModelMeasureReason
SEM-S33%7%15%45%
SEM-T50%10%20%20%
EBSD-S/O24%6%35%35%

Numerical tolerances. We follow task-specific procedures to confirm numerical tolerances.

  • SEM-S: Five-fold validation P90 errors on an independent development set: 0.019 (pores), 0.009 (cracks), 0.042 (semi-molten particles).
  • SEM-T: Maximum of instance-specific Persistence MAE and one-pixel area fraction.
  • EBSD-S/O: Maximum of reconstruction-error P75 and expert-defined floors.

Sensitivity analysis. We tested alternative weights and tolerances across all 15 models and 64 instances. The Top-3 model set remained unchanged in all tests, supporting ranking robustness despite possible changes in absolute scores.

ConfigurationSpearman ρ ↑
Equal capability weights0.9750
Capability weights ±20%0.9978
Numerical tolerances1.0000

Q2: How is executable-code correctness evaluated, and does the high Model pass rate limit its discriminative ability?

A: Our verification assesses functional correctness beyond executability, and the results reveal substantial task-dependent differences in discrimination.

Functional verification. Submitted programs are tested on controlled mask inputs, including sparse foreground, full-image normalization, and empty masks. Tests also check consistency between reported measurements and masks. EBSD-O additionally verifies valid-region normalization and grain-selection constraints. Missing or invalid artifacts receive zero for affected checks.

Full-pass rates across 15 models:

TaskFull-pass rate
SEM-S95.00%
SEM-T91.11%
EBSD-S72.67%
EBSD-O15.33%

While SEM tasks approach saturation, EBSD-S and particularly EBSD-O remain discriminative, demonstrating that Model evaluation tests scientific computation rather than execution alone.

Q3: What inputs does the GPT-5.4 judge receive, and how is potential evaluator/model-family bias controlled?

A: We clarify the judge inputs and provide cross-family and human validation, both showing strong agreement with GPT-5.4.

Judge inputs. For all four tasks, GPT-5.4 receives the agent's scientific analysis, expert-annotated reference conclusions (ground truth), and task-specific scoring rubrics. Raw images, masks, numerical outputs, and code are not directly provided to the Reason judge. Complete prompts and protocols will be open-sourced.

Cross-family validation. We re-evaluated all 15 agents using Claude Sonnet 5 and Gemini 3.1 Flash Lite under identical rubrics and scoring rules. Both judges preserved the same Top-3 model set. GPT-5.6 Sol ranked first under both alternative judges, while Claude Opus 5 ranked first under GPT-5.4, suggesting limited dependence on the evaluator's model family.

Alternative judge vs. GPT-5.4Spearman ρ ↑Kendall τ-b ↑
Claude Sonnet 50.98570.9429
Gemini 3.1 Flash Lite0.97140.8857

Human validation. We expanded validation from 20 to 100 outputs, independently assessed by two materials experts across all four tasks. Spearman and ICC assess overall Reason scores and rubric-item agreement measures criterion-level consistency. Human–LLM agreement is comparable to inter-expert agreement, further supporting Reason evaluation reliability.

ComparisonSpearman ρ ↑ICC(A,1) ↑Rubric-item agreement ↑
Expert 1 vs. GPT-5.40.8670.85386.05%
Expert 2 vs. GPT-5.40.9330.93890.53%
Expert 1 vs. Expert 20.9130.91884.68%
AuthorsOpenReview ↗

Official Response by Authors to Reviewer 2Y7p (2/2)

Part 2: Validity of SEM-T Evaluation

Q4: Do SEM-T metrics capture meaningful microstructural damage evolution rather than only global image similarity?

A: We appreciate the reviewer's concern that generic image similarity may overlook localized microstructural damage. SEM-T already combines global similarity metrics with four defect-aware measurements, explicitly evaluating quantitative changes in damage extent.

Beyond FSIM and 1−GMSD, SEM-T evaluates:

  • Total defect area: Overall defect coverage.
  • Crack area: Extent of crack-related defects.
  • Pore area: Extent of pore-related defects.
  • Newly formed defect area: Fraction of pixels newly classified as defective relative to the preceding cycle.

These measurements are important because microstructural damage can occupy relatively small image regions. A prediction preserving most of the original image may therefore achieve high global similarity without accurately capturing changes in defects. The combined evaluation distinguishes overall visual fidelity from quantitative damage evolution.

Q5: Can agents outperform a no-change persistence baseline, and what are the remaining limitations of forecasting evaluation?

A: We conducted three independent runs for four representative models on a fixed 32-instance subset (10 SEM-S, 12 SEM-T, 5 EBSD-S, and 5 EBSD-O), sampled without replacement while preserving specimen-level dependencies. The results show relatively small run-to-run variations (standard deviations of 0.73–1.13 points). The same three leading models consistently outperform Gemini 3.7 Flash across all runs, although their internal ordering varies. These findings support the stability of the leading model group under repeated executions within the evaluated subset.

MetricPersistenceClaude Opus 5GPT-5.6 SolGPT-5.5
FSIM ↑0.81050.76920.81870.8087
1−GMSD ↑0.87840.85870.88410.8751
Defect Area MAE ↓0.02080.01580.01490.0176
Crack Area MAE ↓0.01380.01040.01050.0138
Pore Area MAE ↓0.00750.00590.00510.0053
New Defect Area MAE ↓0.04520.01860.02840.0231

Part 3: Dataset Dependencies and Ranking Stability

Q6: Do shared specimens and correlated instances affect confidence in model rankings?

A: We conducted 10,000 specimen-level bootstrap resamples across all 15 models, preserving related SEM-T instances and paired EBSD specimens. The three leading models consistently form a stable Top-3 group, with inclusion frequencies of 100.00%, 99.66%, and 99.96%, respectively. Although their internal rankings may vary across resamples, these results indicate that the identification of the leading model group is robust to specimen-level sampling variability.

ModelOverall95% Bootstrap CITop-3 frequency
Claude Opus 553.38[51.44, 55.26]100.00%
GPT-5.6 Sol53.33[51.00, 55.56]99.66%
GPT-5.552.95[51.52, 54.44]99.96%
Gemini 3.7 Flash49.04[47.11, 50.84]0.01%

Q7: Are rankings stable under repeated agent executions?

A: We evaluated four representative models over three independent runs on a fixed 32-instance subset (10 SEM-S, 12 SEM-T, 5 EBSD-S, and 5 EBSD-O), sampled without replacement while preserving SEM-T specimen groups and paired EBSD samples. The results show relatively small run-to-run variations (standard deviations of 0.73–1.13 points). The same three leading models consistently remain ahead across all runs, although their internal ordering varies, supporting the stability of the leading model group within the evaluated subset.

ModelRun 1Run 2Run 3Mean ± SD
Claude Opus 552.0750.2251.7451.34 ± 0.99
GPT-5.6 Sol52.2550.3551.4251.34 ± 0.95
GPT-5.552.6253.4454.0853.38 ± 0.73
Gemini 3.7 Flash48.8148.0946.5947.83 ± 1.13

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their thoughtful evaluations and constructive feedback. We appreciate their recognition of MatMicroBench's contributions. Reviewer 2Y7p highlighted the value of our experimentally acquired SEM and EBSD data and described the artifact-level evaluation as "well motivated", emphasizing its ability to diagnose agent failures beyond final answers. Reviewer P7s3 recognized the "grounded experimental study" and its "diagnostic insights", particularly the identification of capability bottlenecks and harness effects. Reviewer tnGG acknowledged the "very comprehensive experimental results" and the clarity of the paper.

We are also grateful for the reviewers' insightful suggestions, which have provided valuable opportunities to further enhance the transparency, reliability, and scientific rigor of MatMicroBench. Building on these suggestions, we have provided the following clarifications and complementary analyses:

  1. Evaluation transparency and reliability. We clarified scoring configurations, numerical tolerances, and executable-code verification procedures. We also conducted cross-family judge validation and expanded human evaluation to 100 outputs, confirming strong agreement across evaluators.
  2. Ranking robustness. We performed scoring sensitivity analyses, specimen-level bootstrap resamples, and repeated agent executions. These results consistently preserved the leading model group under the tested conditions.
  3. Scientific evaluation and baselines. We conducted additional SEM-T persistence comparisons and SEM-S segmentation experiments with Otsu, MatSAM, and SAM 3, further supporting the importance of defect-aware measurements and semantic microstructure analysis.
  4. Implications for agent development. We clarified how the identified capability bottlenecks can guide future domain-specific skill augmentation and trajectory-level optimization.

We will incorporate the relevant clarifications and supporting results into the revised manuscript and appendices, and release the evaluation configurations and protocols to facilitate reproducibility.

We thank the reviewers again for their valuable feedback, which has helped us further strengthen the presentation and validation of MatMicroBench.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.