acceptodds
Under review as a conference paper at ICLR 2027

Struct-R1: Material Structure Search with Multi-Fidelity Measurement in Agentic Loops

Abstract

Materials discovery requires a research loop in large structure space with multi-tier measurements with different fidelity and costs, from machine-learned surrogates to DFT calculations and experiments. Manual exploration can be time-consuming, while statistical search methods and existing agent designs struggle to achieve high search efficiency with trade-off between fidelity and cost. We present Struct-R1, a self-evolving agentic framework with two coupled loops. The inner search loop performs physics-grounded reasoning and perturbation proposal, and manages cross-tier relationships to improve search efficiency. The outer evolution loop reflects on individual search steps and converts their feedback into reusable knowledge. Through the two loops, Struct-R1 progressively refines both where to explore and how to allocate search budget on the measurements. We evaluate Struct-R1 on representative material-property search tasks, such as Raman and phonon properties, and further validate it using real research data for well-studied materials to assess its effectiveness in realistic scientific settings.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI for Science, Automatic Research Loop, Material Science, Material Structures

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XkgqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Focus on material search, this work introduces an agentic framework to refine a material structure targting on a specific design requirements. The major novelty lies in two aspects: multi-fidelity material reasoning design (inner loop) and evolution loop (outer loop) with self-reflection. Evaluations on two tasks (ie, search task and stable structure discovery benchmark) show its superiority compared to traditional methods such as MLIP, and SOTA LLMs such as GPT 5.6. Finally, the analysis and case study further demonstrate its effectiveness.

Critical strengths
  • Presentation: The work is well structured and clearly written; the challenges are clearly presented, and the solutions are stated to reasonably address them.
  • Novelty: This work not only introduces an effective multi-fidelity framework for material structure discovery, but also proposes an efficient reflection method for agent harnessing, which may further generalize this method to other scientific domains.
  • Evaluation: The experiments are elaborated to demonstrate its strong superiority. For example, it improves the overall success rate from about 50% to 90% on search performance.
Critical weaknesses
  • Novelty: Although the overall framework is novel, the major solutions, ie, multi-fidelity solutions that combine MLIP and DFT, and self-reflection solutions that utilize historical trajectory (records in this work), are not new. The authors should compare and discuss with some similar works, including but not limited to examples such as multi-fidelity work [1,2] and self-reflection on material discovery [3].
  • The authors demonstrate its effectiveness in two benchmarks, ie, computational Raman and phonon property search tasks and retrospective literature case studies on well-studied materials (eg, Si, etc), but they claim a broader effectiveness for general, autonomous materials discovery and arbitrary multi-perturbation (strain, vacancy, substitution, and compositional doping) structure search (for example, the authors claim "General "materials structure discovery" or ""combinatorially large" perturbation space"). The authors should narrow down their claims (and research focus) or include much more comprehensive experiments.

[1] Hernández-García, A., Saxena, N., Jain, M., Liu, C. H., & Bengio, Y. "Multi-Fidelity Active Learning with GFlowNets". 2024. TMLR.

[2] Chen., Zhan ., et al. "PhyVer: Physics-Grounded Material Claim Verification with Multi-Fidelity Physical Evidence. ACL. 2026.

[3] Abhyankar,et al.. "LLEMA: Evolutionary search with LLMs for multi-objective materials discovery." ICLR. 2026.

Questions for authors

Try to address the weaknesses with concrete experiments or clear revisions.

5 comments
AuthorsOpenReview ↗

Response to the questions - part 1

We thank the reviewer for the detailed and constructive feedback. We will add new comparison results and adjust the imprecise claim in the paper. Below are the responses.

Weakness 1: Baselines and related work

We thank the reviewer for pointing out these related works. We believe our core contribution lies in the agent harness design for materials search, which enables efficient utilization and continuous evolution of cross-fidelity knowledge to balance search accuracy and measurement cost. Existing multi-fidelity methods, such as MF-GFN [1], rely on statistical or learned models that can be unreliable when low-fidelity measurements are biased or uninformative. Meanwhile, LLM-based approaches such as PhyVer [2] and LLEMA [3] focus on claim verification and evolutionary materials discovery, respectively, without explicitly enabling the agent to maintain and update cross-fidelity relationships for cost-aware search decisions. We also added new baselines experiments, using the same results in Question 5 in Review ehTF, including a transferred version of LLMatDesign, LLEMA, MF-MES, and a simple ReAct Agent. MF-GFN [1] requires training a GFlowNet policy, while Struct-R1 and all other baselines do not train a neural policy. We therefore did not compare with it directly. Instead, we adopted MF-MES, the cost-aware multi-fidelity acquisition function used by MF-GFN.

LLMatDesign and LLEMA were designed for composition search and do not directly support our tasks. Following their released code, we adapted their core search strategies to our tools, budgets, and questions. LLMatDesign proposes one modification per step and uses self-reflection and search history to guide subsequent steps. LLEMA evolves a population using successful and unsuccessful candidates, and generates five new candidates per iteration. MF-MES is a cost-aware multi-fidelity Bayesian optimization baseline that uses a Gaussian process to model target-property errors across configurations, strains, and fidelity tiers. It selects the next candidate and measurement fidelity based on expected information gain per unit time cost, and updates the model after each measurement. We also build a ReAct agent that can call MLIP, MLIP pre-relax and DFT as tools. All experiments use GPT-5.6 Terra and the same tasks as Table 2. We will report details of these new baselines in the revision.

Method (GPT-5.6 Terra)RamanElasticChargeDecayBornOverall
LLMatDesign5 / 7.919 / 4.3100 / 5.020 / 5.1100 / 3.840% / 5.6
LLEMA42 / 13.272 / 9.490 / 7.555 / 9.685 / 7.067% / 10.0
MF-MES78 / 8.685 / 7.0100 / 4.490 / 6.8100 / 5.788% / 6.8
ReAct98 / 9.182 / 10.2100 / 9.5100 / 11.1100 / 8.694% / 9.6
Struct-R1 (GPT-5.6 Terra, Table 2)100 / 5.192 / 5.2100 / 4.5100 / 5.3100 / 3.898% / 4.9

LLMatDesign performs sequential refinement and can be misled when MLIP feedback is biased, so the accuracy is low despite the lower DFT call numbers. LLEMA is more robust because it maintains a population of candidates, but evaluating five candidates per generation increases DFT cost. MF-MES achieves the highest success rate among non-LLM baselines through cost-aware fidelity selection, but performs less effectively with biased or uninformative MLIP feedback. A ReAct agent with the same tools can reach better balance, but because of the lack of relationship records between the tiers, it still has room for improvement.

Please notice that LLMatDesign and LLEMA were not designed for the same multi-fidelity setting as Struct-R1, so our goal is not to show that these methods fail. Instead, the comparison highlights the need for a dedicated harness system that explicitly manages measurement cost, fidelity, and cross-tier relationships for the multi-tier tools in the materials science research.

AuthorsOpenReview ↗

Response to the questions - part 2

Weakness 2: Scope of the generality claim

We appreciate the reviewer’s comment on the generalizability of Struct-R1. We agree that the scope of our empirical validation should be stated more precisely. Our current experiments cover five target properties (Raman shift, elastic energy, charge transition level, phonon decay, and Born effective charge) on Si, Ge, GaAs, and MgO, with point defects and uniaxial strain as perturbations. They also include the Matbench Discovery test and two case studies. We will revise the Abstract, Introduction, and Conclusion so that the claims match this empirical scope.

Part of what broader search requires is exercised in our experiments. Struct-R1 models the role of the cheap tier through structured tier records. In our experiments, this role differs across tasks. Cheap tiers can be used directly for measurement calibration, or serve only as trend indicators, or be uninformative or even unavailable in some cases. The tier records and self-evolution let the agent identify and apply the right role for each task and optimize the search strategies. We also added transfer experiments in our response to Question 2 of Reviewer VkDf, where search hints learned on one material or property were frozen and applied to unseen tasks. In several cases, the transferred hints improved success rates and reduced DFT costs, showing the potential of such learned experience to generalize across materials and properties. Struct-R1 combines agent reasoning with this harness design, so it is designed to handle these situations as the search space broadens.

We agree that such generalization requires further systematic validation. Extending the evaluation to more complex materials, broader property targets, and larger perturbation spaces remains an important direction for future work. We will add a discussion of these limitations and future directions in the revised paper.

We thank the reviewer again for the valuable and constructive feedback. We would be happy to provide further clarification or additional experimental results if the reviewer has any remaining questions.

AuthorsOpenReview ↗

Follow-up on the rebuttal responses

Dear Reviewer Xkgq,

Thank you again for your constructive feedback. We have added further comparisons and revised the manuscript to clarify our contributions and empirical scope. We sincerely hope these revisions can addresse your concerns.

We would greatly appreciate hearing if you have any remaining concerns. As the discussion period is already more than halfway through, we still have time to provide further experiments or clarification if needed.

Thank you again for your time and consideration.

Best regards,

The Authors

Reviewer_VkDfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Struct-R1, an agent framework for material structure search equipped with multi-fidelity measurements. The agent decides the next action for structural perturbations, and then selects between cheaper evaluations and DFT calculations, which provide higher fidelity at a higher cost. The framework records the relationships between measurement tiers and reflects on past search trajectories to update its search and selection strategy. Experiments on material-property search, stable-structure discovery, and research-data case studies provide evidence that Struct-R1improves search success over several baselines, and reduces DFT usage.

Critical strengths
  1. The paper targets a very concrete and meaningful problem, since DFT calculations are expensive and cheaper models may lead to inaccurate feedback. Also, Struct-R1 achieves strong results on the main benchmark in the paper's experiments. So I believe the authors do propose a framework that could be useful for materials search under limited measurement budgets.

  2. The experiments cover several different materials and target properties. The case studies offer insights into how Struct-R1 could operate in practical scientific problems.

Critical weaknesses

The specific task of the paper is quite meaningful, and the Struct-R1 provides interesting insights. But I still have the following two main concerns:

  1. The main LLM baselines in the experiments are, from my perspective, too weak. These LLM baselines don't use any iterative feedback, which is a natural result that they will obtain weaker results than Struct-R1. Several existing material agents such as LLMatDesign[1] and LLEMA[2] already support feedback-driven search.

  2. The efficiency and transferability of the learned experience are not clear. Struct-R1 does reduce DFT calls, but it's unclear whether Struct-R1 requires a higher cost of obtaining and updating experience. Also, the relationships between measurement tiers may change across materials and properties, resulting in repeated calibrations.

References:

[1]. Jia, Shuyi, Chao Zhang, and Victor Fung. "Llmatdesign: Autonomous materials discovery with large language models." arXiv preprint arXiv:2406.13163 (2024).

[2]. Abhyankar, Nikhil, Saaketh Desai, and Chandan Reddy. "LLEMA: Evolutionary search with LLMs for multi-objective materials discovery." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors
  1. How does Struct-R1 compare with stronger agents with iterative feedback, such as LLMatDesign and LLEMA, using the same LLM and tools under comparable budgets?

  2. How much of the learned experience can be reused on new materials or properties? Do the savings from fewer DFT calls outweigh the cost of forming the new experience?

8 comments
AuthorsOpenReview ↗

Response to the questions - part 1

We thank the reviewer for the detailed and constructive feedback. We have added more baselines and analysis of the extra costs.

Weakness 1 & Question 1: Stronger baselines

The question is similar to Question 5 in Review ehTF. so we use the same new experiment results as the answer. We added new baselines including a transferred version of LLMatDesign and LLEMA, MF-MES, and a ReAct Agent. LLMatDesign and LLEMA were designed for composition search and do not directly support our tasks. Following their released code, we adapted their core search strategies to our tools, budgets, and questions. LLMatDesign proposes one modification per step and uses self-reflection and search history to guide subsequent steps. LLEMA evolves a population using successful and unsuccessful candidates, and generates five new candidates per iteration. MF-MES is a cost-aware multi-fidelity Bayesian optimization baseline that uses a Gaussian process to model target-property errors across configurations, strains, and fidelity tiers. It selects the next candidate and measurement fidelity based on expected information gain per unit time cost, and updates the model after each measurement. We also build a ReAct agent that can call MLIP, MLIP pre-relax and DFT as tools. All experiments use GPT-5.6 Terra and the same tasks as Table 2. We will report details of these new baselines in the revision.

Method (GPT-5.6 Terra)RamanElasticChargeDecayBornOverall
LLMatDesign5 / 7.919 / 4.3100 / 5.020 / 5.1100 / 3.840% / 5.6
LLEMA42 / 13.272 / 9.490 / 7.555 / 9.685 / 7.067% / 10.0
MF-MES78 / 8.685 / 7.0100 / 4.490 / 6.8100 / 5.788% / 6.8
ReAct98 / 9.182 / 10.2100 / 9.5100 / 11.1100 / 8.694% / 9.6
Struct-R1 (GPT-5.6 Terra, Table 2)100 / 5.192 / 5.2100 / 4.5100 / 5.3100 / 3.898% / 4.9

LLMatDesign performs sequential refinement and can be misled when MLIP feedback is biased, so the accuracy is low despite the lower DFT call numbers. LLEMA is more robust because it maintains a population of candidates, but evaluating five candidates per generation increases DFT cost. MF-MES achieves the highest success rate among non-LLM baselines through cost-aware fidelity selection, but performs less effectively with biased or uninformative MLIP feedback. A ReAct agent with the same tools can reach better balance, but because of the lack of relationship records between the tiers, it still has room for improvement.

Please notice that LLMatDesign and LLEMA were not designed for the same multi-fidelity setting as Struct-R1, so our goal is not to show that these methods fail. Instead, the comparison highlights the need for a dedicated harness system that explicitly manages measurement cost, fidelity, and cross-tier relationships for the multi-tier tools in the materials science research.

AuthorsOpenReview ↗

Response to the questions - part 2

Weakness 2 & Question 2: Transferring and Costs

We thank the reviewer for this important question. To evaluate the transferability and cost of learned experience, we conducted additional experiments in which search hints learned from one set of tasks are frozen and applied to unseen tasks involving different materials or properties.

What experience can be transferred?: Struct-R1 learns two types of experience: (1) search hints, such as how to use cheap measurements to localize promising regions and select reference DFT measurements for calibration; and (2) material-specific relationships between measurement tiers, such as the sign and magnitude of MLIP-to-DFT corrections. The former can generalize across materials and properties, while the latter requires recalibration for new tasks. The learned hints explicitly record these limitations to avoid transferring unreliable numerical relationships.

Evidence of transfer across materials: We let the agent learn search hints from 10 Si training tasks, then freeze them and directly apply them to 20 unseen tasks on Ge, GaAs, and MgO. The training and test tasks use disjoint hidden perturbations. No evolution is performed on the target materials. We compare against the same test tasks without hints.

ModelSource → TargetNo hintsTransferred hints
GPT-5.6 TerraSi → Ge elastic19/20 · 7.820/20 · 6.7
GPT-5.6 TerraSi → GaAs elastic16/20 · 7.317/20 · 5.5
GPT-5.6 TerraSi → MgO elastic19/20 · 2.620/20 · 1.9
GPT-5.2Si → Ge elastic7/20 · 10.010/20 · 10.3
GPT-5.2Si → GaAs elastic12/20 · 8.617/20 · 5.4
GPT-5.2Si → MgO elastic19/20 · 7.319/20 · 6.3

Cross-property transfer. We also evaluate whether search hints learned from one property can be transferred to another property of the same material. We consider three transfer settings on Ge, covering elastic, Raman, and charge-level properties. Each contains 20 test tasks. Each entry reports solved tasks / average DFT calls per task. We compare search performance with and without transferred hints using two LLM backbones.

ModelSource → TargetNo hintsTransferred hints
GPT-5.6 TerraGe Elastic → Ge Charge20/20 · 4.420/20 · 3.1
GPT-5.6 TerraGe Elastic → Ge Raman20/20 · 4.120/20 · 4.0
GPT-5.6 TerraGe Raman → Ge Elastic19/20 · 7.820/20 · 4.7
GPT-5.2Ge Elastic → Ge Charge20/20 · 12.320/20 · 9.9
GPT-5.2Ge Elastic → Ge Raman19/20 · 9.118/20 · 8.0
GPT-5.2Ge Raman → Ge Elastic7/20 · 10.012/20 · 10.9

These results suggest that learned search experience can be reused across certain settings, improving overall search efficiency and effectiveness. However, cross-property transfer is less general than cross-material transfer, as different properties sometimes involve distinct physical mechanisms, while the same property often shares similar physical relationships across materials. We will clarify this in the revised version.

Transfer to held-out materials and properties. The preceding experiments evaluate transfer using a small number of training tasks. We further investigate whether learned experience transfers when using larger training sets spanning multiple materials or properties on GPT-5.6 Terra. For cross-material transfer, we evolve the harness on Si and Ge and test it on GaAs and MgO, covering all available properties. For cross-property transfer, we evolve on Raman, charge, and decay tasks and test on elastic and Born tasks across all four materials. In both settings, the evolution and test sets are disjoint, and the evolved harness is frozen during testing.

Evolution → TestNo EvolutionWith Evolution
Si, Ge → GaAs, MgO95.8% / 4.996.7% / 4.1
Raman, Charge, Decay → Elastic, Born94.0% / 4.999.0% / 4.1

The results show that evolution improves success rates and reduces DFT calls in both settings. Cross-material transfer improves success from 95.8% to 96.7%, while cross-property transfer improves success from 94.0% to 99.0%. Both settings reduce DFT calls from 4.9 to 4.1, approximately 16%. These results suggest that the learned experience can be reused beyond the materials and properties encountered during evolution.

AuthorsOpenReview ↗

Response to the questions - part 3

Does the extra cost of evolution outweigh the benefits: To evaluate whether the benefits of evolution justify its learning cost, we conduct an experiment using Si elastic search. For each model, we run 10 training tasks and generate one reflection after each task to learn reusable search hints. We then freeze these hints and apply them to 80 elastic tasks (Si, Ge, GaAs, and MgO). We compare search performance with and without hints under the same measurement budget.

ModelTasksSolved (No hints → Hints)DFT calls (No hints → Hints)Reflection share of cumulative API cost
GPT-5.2, online learning1010/10898.2%
GPT-5.2, Si elastic2019 → 18191 → 166 (−25)2.5%
GPT-5.2, Other materials, elastic6038 → 46515 → 438 (−77)0.82%
GPT-5.6 Terra, online learning1010/102312.5%
GPT-5.6 Terra, Si elastic2020 → 2059 → 46 (−13)5.0%
GPT-5.6 Terra, Other materials, elastic6054 → 57353 → 282 (−71)1.1%

The learning stage requires 89 DFT calls for GPT-5.2 and 23 for Terra, with additional reflection costs of only 0.06 USD and 0.05 USD, respectively. Across the 80 subsequent tasks, the learned hints reduce DFT calls by 102 for GPT-5.2 and 84 for Terra, respectively. In both cases, the DFT savings exceed the initial learning cost.

Moreover, Struct-R1 follows an online evolution process, where search experience is collected and reflected upon during material search tasks rather than through a separate learning phase. Therefore, the DFT evaluations used to generate search experience are already part of the normal search workload, and only the lightweight reflection calls introduce additional learning overhead. As search hints can be reused partially across tasks, their benefits can accumulate.

Transferrable contents: The learned hints contain both material-specific calibrations and reusable search hints. For example, GPT-5.2 learns from Si elastic tasks that "reference − cheap ≈ +0.009 − 0.091·strain for the host," while also recording the hint such as "scan with the cheap tier to find the side of the energy minimum, anchor once with DFT, then confirm; do not extrapolate the correction across zero strain." The numerical calibration is specific to Si, but those search hints can be transferred to other materials.

We have provided additional experimental details and ablation studies in our response to Reviewer ehTF. We would be happy to provide further clarifications or additional experimental results if the reviewer has any remaining questions or concerns.

Reviewer_ehTFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Struct-R1, an LLM agent for materials structure search that combines a cheap MLIP tier (MACE) with expensive DFT. The agent keeps natural-language "tier records" describing how the cheap tier relates to DFT, and uses them to decide which tier to query at each step. An outer reflection loop then updates these records, along with search hints and DFT settings, based on past search trajectories. The authors evaluate it mainly on a new benchmark where the agent has to recover a hidden perturbation (point defect plus uniaxial strain) in Si, Ge, GaAs or MgO from a target property. With GPT-5.6 Terra, they report 98% success with about 5 DFT calls, compared with 61% for multi-fidelity BO. They also include a Matbench Discovery experiment and two case studies, on MAPbI3 and HfTe5, that replay published data.

Critical strengths

The paper tackles a well-motivated problem. Multi-fidelity search where the cheap tier's reliability changes across materials and regions is a real bottleneck in computational materials science. Fig. 3 makes a convincing case that a fixed cross-tier model can be confidently wrong: AR1/NARGP-style fits mislocate the SnTe transition, and the GaAs cheap tier carries no information. Letting an agent keep a revisable, qualitative belief about the cheap tier ("trend only," "transition location shifted," "exclude") is a plausible way to get robustness that standard MFBO lacks. This is the main reason the paper is interesting.

The paper proposes a verifiable, DFT-backed evaluation setup. The hidden-perturbation benchmark checks success against DFT ground truth, and it is grounded in real (if modest) DFT effort: about 1,000 cached evaluations and more than 8,000 CPU-core-hours. Releasing it as a shared, cached oracle would be useful to the community, and the design is better than evaluating only against surrogate feedback, as prior LLM materials agents do.

Critical weaknesses

My main concern is that the benchmark and protocol don't support the headline result of 98% success versus 61% for MFBO. Each task is a host crystal, one of about four defect configurations, an axis in {x, y, z}, and a scalar strain within ±1.5–2.5%. That's really around a dozen one-dimensional line searches. In a space that small, a strong LLM can get far on standard physics it already knows, such as deformation potentials for the Si Raman shift. So I can't tell how much of the gain comes from the multi-fidelity reasoning and how much from prior knowledge. The hidden perturbation stops the model from recalling a specific answer, but not from using general structure–property knowledge. I also couldn't work out whether the agent can query arbitrary perturbations or only points in the precomputed DFT/MLIP cache. If it's only the cache, success depends heavily on grid resolution and tolerance, and neither is stated. The evolution loop is run on Ge elastic-energy tasks, but the paper doesn't say whether Table 2 uses an evolved harness. If it does, it doesn't say whether the evolution tasks overlap with the test tasks. Without a clean split, the evolved hints may just be fitted to the benchmark. Finally, several Table 2 entries can't come from a single pass over the stated task counts. For example, 3% and 1% on 20 tasks, and 87% and 97% on 60 tasks, all imply fractional numbers of solved tasks. So the cells must be averages over repeated runs, yet the number of runs, the seeds, and any variance are never reported. With only 20 Decay and 20 Born tasks, differences like 80% versus 90% mean little without confidence intervals. The identical MFBO and NARGP entries in the Charge and Born columns also make me suspect a transcription error.

My second concern is that the cost accounting and baselines are too thin for a paper framed as cost-aware. Section 3.5 says costs are normalized by computation time, but Table 2 reports only DFT call counts. That treats a 64-atom charged-defect supercell, an 8-atom DFPT Born-charge calculation, and an MLIP-pre-relaxed DFT run as equal. It also leaves out cheap-tier calls and LLM inference cost entirely. I'd need CPU-hours plus token cost to believe the efficiency claims. The baselines are classic AR1/NARGP, grid search, and two weak LLM baselines. There's no comparison with the LLM materials agents the paper cites (LLMatDesign, MatEvolve, LLEMA), with a plain ReAct agent using the same tools, with LLM-prior plus BO hybrids, or with stronger cost-aware multi-fidelity BO such as MF-MES or MISO. There's also no ablation of the tier records or of the two reflection stages, so the gap can't be traced to any of the paper's contributions. The Matbench Discovery experiment, which I'd hoped would address this, actually cuts the other way. With MACE-MPA-0, simple MLIP ranking scores 191 against 181 to 189 for Struct-R1, and evolution adds almost nothing with MEGNet (122 to 123). The metric in Table 3 and the "SFBO 61 (>20)" entry aren't defined, and the possibility that public WBM data appears in LLM pretraining data isn't discussed.

Questions for authors
  • Can the agent query arbitrary perturbations, or only points on the precomputed DFT/MLIP grid? What is the grid resolution, and what tolerance defines success for each property? How many distinct perturbations satisfy the tolerance for a typical task?
  • Were the Table 2 results obtained with an evolved harness? If so, which tasks were used for evolution, and are they disjoint (in host material and property) from the evaluation tasks? Please report Table 2 with a frozen, un-evolved harness as well.
  • How many independent runs or seeds make up each Table 2 cell (the fractional task counts imply more than one)? Please give standard deviations or confidence intervals, and confirm the identical NARGP/MFBO entries.
  • Please report total cost in CPU-hours (DFT plus MLIP) and LLM token cost for each method, not only DFT call counts. How is a DFT call on an MLIP-pre-relaxed structure counted?
  • Please add an ablation of tier records and of each reflection stage on the full Table 2 benchmark, and a comparison with at least one existing LLM materials agent (e.g., LLMatDesign) and a plain ReAct agent using the same tools.
  • What exactly does Table 3 measure, and what does "SFBO 61 (>20)" mean? How do you rule out LLM memorization of WBM/Matbench Discovery?
9 comments
AuthorsOpenReview ↗

Response to the weaknesses - part 1

We thank the reviewer for the detailed and constructive feedback. We added more details, analysis and new results to respond to these weaknesses.

Weakness 1-1: Concerns about domain knowledge recalling

Thanks for bringing this concern up, this is a central point of benchmark design. We want to answer it in two parts with extra experiment results. First, we introduce a masking control to test whether the agent's performance depends on recalling material-specific knowledge, and observe comparable results. Second, the task itself is not completely the textbook problem.

New ablation with material name masking: we change the input question of the benchmark. We masked the material name and DFT settings in both the task questions and thus decision agent’s search context. The decision agent only gets the numerical feedbacks from the measurement engine. DFT script generation retains the necessary chemistry in a separate context that is never exposed to the decision step of the agent. For GPT-5.6 Terra, the results show that masking does not materially affect performance:

Success rate / DFT callsSiGeGaAsMgOOverall
Original (Table 2)100% / 4.798% / 5.493% / 5.698% / 4.198% / 4.9
Masked99% / 4.798% / 4.798% / 5.8100% / 4.599% / 4.9

Doped structures are not completely textbook: Each task involves a given material structure with multiple possible defect configurations. These configurations and their corresponding properties are not widely documented in the literature, and the specific numerical values also depend on DFT settings such as the pseudopotential. We further tested whether the model could predict the configuration and direction of property changes in a one-shot setting without measurement feedback. The results are summarized below:

AccuracySiGeGaAsMgOOverall
Configuration correct28%28%33%42%32%
Direction correct28%13%20%15%20%
Both correct10%7%3%7%7%

Configuration and direction accuracy is relatively low, while the joint accuracy is lower. These results suggest that prior knowledge alone does not reliably identify the correct configuration and search direction; measurement feedback and iterative reasoning are needed.

Weakness 1-2 & Question 1: Search space configuration details

The framework allows arbitrary configuration and strain queries. For evaluation, we use discrete grids shared across all methods to limit computation costs. The maximum strain ranges and step sizes are at most (±2.5%, 0.25%) for Si and MgO, and (±1.5%, 0.15%) for Ge and GaAs. With 4–5 atom-doping configurations and 3 axes, this gives approximately at most 240-300 structures per search space. Our materials science experts checked and set these ranges and step sizes, ensuring physically meaningful structures.

The numerical tolerances for each target property are summarized below.

PropertyRamanElasticChargeDecayBorn
Tolerance0.3 cm⁻¹10 meV25 meV0.3 cm⁻¹0.05 e

These search ranges, grid resolutions, and numerical tolerances were selected in consultation with materials science experts to balance computational measurement costs with scientific accuracy. Across the benchmark questions, the median number of candidates satisfying the tolerance is only 3–6 per question, requiring precise exploration to identify valid solutions.

AuthorsOpenReview ↗

Response to the weaknesses - part 2

Weakness 1-3 & Question 2: Evolution and datasets

Table 2 does not include evolution. The agent's initial records provide basic information about MLIP and DFT feedback, but does not use additional reflection stages or search hints learned from previous tasks. We evaluate evolution separately through the ablation studies in Weakness 2-2.

Weakness 1-4 & Question 3: Evaluation seed and Table 2 results

Thank you for highlighting the missing evaluation details and inconsistencies in Table 2. We have checked the run logs, corrected two transcription errors, and conducted additional experiments to assess sensitivity to question seeds.

Table corrections: In the submitted table, some entries in the Single shot and Plan & execute rows mistakenly reported the number of solved questions rather than percentages. Their overall success rates were also incorrectly reported as 4% and 46%, respectively, instead of the correct 7.7% and 64.2%. The 87% and 97% entries on the 60 charge-level tasks correspond to 52/60 and 58/60, rounded to integers. We have corrected these reporting errors and rechecked the results against the run logs.

Evaluation seeds and additional runs: Each entry in the original Table 2 comes from a single run with random seed 0. To evaluate sensitivity, we evaluated every method on two additional random seeds 1 and 2. We will report these details in the revised paper. The new Table 2 is as below:

MethodRamanElasticChargeDecayBornOverall
Grid9±1 / 9.7±0.147±3 / 11.4±0.586±1 / 5.5±0.330±10 / 8.7±0.460±0 / 9.8±0.444±1% / 9.2±0.2
MLIP only0±1 / 0.0±0.015±4 / 0.0±0.00±0 / 0.0±0.00±0 / 0.0±0.00±0 / 0.0±0.05±1% / 0.0±0.0
MFBO (AR1)65±4 / 7.2±0.359±7 / 7.6±0.768±9 / 7.6±0.855±5 / 8.1±0.482±8 / 7.0±0.764±4% / 7.5±0.4
NARGP30±5 / 10.3±0.473±8 / 6.8±0.768±9 / 7.6±0.847±8 / 8.8±0.682±8 / 7.0±0.757±7% / 8.2±0.6
Single shot0±1 / 0.0±0.07±3 / 0.0±0.016±8 / 0.0±0.07±8 / 0.0±0.013±8 / 0.0±0.07±2% / 0.0±0.0
Plan & execute56±12 / 9.3±1.380±8 / 10.9±2.471±3 / 9.6±0.418±24 / 7.7±3.870±28 / 7.0±2.665±4% / 9.5±0.8
ReAct99±1 / 9.0±0.279±3 / 10.0±0.1100±0 / 9.4±0.2100±0 / 10.9±0.5100±0 / 9.0±1.193±1% / 9.5±0.1
Struct-R1 GPT-5.6 Terra100±1 / 5.0±0.195±4 / 4.6±0.6100±0 / 4.3±0.2100±0 / 5.5±0.2100±0 / 3.9±0.298±1% / 4.7±0.2
Struct-R1 GPT-5.280±1 / 9.3±0.370±1 / 8.9±0.293±4 / 12.8±0.373±13 / 9.2±0.597±3 / 12.2±0.581±2% / 10.2±0.2
Struct-R1 GPT-4o18±3 / 7.0±0.631±3 / 6.6±0.293±4 / 10.6±0.723±13 / 4.5±0.990±5 / 10.5±0.845±1% / 7.8±0.1
Struct-R1 Claude Sonnet-595±3 / 5.3±0.288±6 / 3.9±0.2100±0 / 5.0±0.180±15 / 5.2±1.3100±0 / 5.4±0.393±3% / 4.8±0.1
Struct-R1 Claude Opus-590±3 / 3.9±0.191±4 / 3.7±0.498±2 / 3.7±0.295±5 / 4.4±0.295±9 / 3.7±0.393±1% / 3.8±0.1

Identical MFBO and NARGP results. In the original single-run table, MFBO and NARGP produced identical results on Charge and Born tasks. For these properties, MLIP cannot directly predict the target values and is used only for structural pre-relaxation before DFT evaluation. Consequently, neither method has direct low-fidelity property measurements to model the cross-tier relationship. In our experiments, the two methods selected same DFT reference points and obtained identical results for these tasks. We will clarify this behavior in the revised manuscript.

AuthorsOpenReview ↗

Response to the weaknesses - part 3

Weakness 2-1 & Question 4: CPU-hour and token cost evaluation

We agree that DFT call counts alone do not fully capture computational cost. In the original paper, the DFT call we reported only counted full DFT evaluations and used this count as a cost proxy. We now additionally report measured compute costs, calculated as the actual wall-clock time of all the DFT calculation and MLIP-based calculation multiplied by the number of CPU cores used, along with LLM reasoning time and token costs. For a DFT call on an MLIP-pre-relaxed structure, we include both the time spent on MLIP pre-relaxation and the time spent on the subsequent DFT calculation of the specified property. It is worth noting that stronger models generally consume fewer total tokens because they reach the target in fewer interaction rounds and optimization iterations. We will incorporate these results into the revised manuscript. Details of new baselines are introduced in Weakness 2-2. Below are the results across all the task types:

MethodSuccessDFT callsMLIP callsDFT wall (min)MLIP wall (min)LLM (min)Total wall (min)DFT core-hMLIP core-hTokens / task (in / out)API cost / task
Grid44%9.370.01950.00.00195179.40.00––
MLIP only4%0.0028.0062.20.00620.01.04––
MFBO (AR1)61%7.7119.414543.10.00188128.70.72––
NARGP51%8.7619.417543.10.00219158.20.72––
MF-MES88%6.7711.814226.00.00168134.30.43––
Single shot (Terra)8%0.000.000.00.100.10.00.00181 / 318$0.004
Plan & execute (Terra)64%9.0430.319177.90.97270176.11.300.9k / 5.8k$0.072
LLMatDesign (Terra)40%5.6022.512162.02.77186114.21.0331.2k / 12.34k$0.211
LLEMA (Terra)67%9.980.02140.00.11214196.80.000.7k / 0.49k$0.007
ReAct (Terra)94%9.635.719014.90.72206166.10.256.4k / 2.07k$0.038
Struct-R1 · GPT-4o46%7.7819.914253.40.80196104.50.8963.8k / 2.79k$0.187
Struct-R1 · GPT-5.283%10.0912.619735.80.36233163.00.6038.9k / 0.30k$0.072
Struct-R1 · Sonnet-590%4.709.69429.62.7812681.90.4924.8k / 15.70k$0.207
Struct-R1 · Opus-592%3.904.18211.40.919470.50.1911.4k / 2.18k$0.112
Struct-R1 · GPT-5.6 Terra98%4.884.810313.50.5911791.10.228.6k / 1.84k$0.039

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Revised Manuscript

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their constructive feedback and detailed suggestions. We have provided point-by-point responses to the reviewers' concerns and prepared a revised manuscript. We summarize the major responses and revisions below.

  1. Stronger baselines. We added comparisons with LLMatDesign, LLEMA, MF-MES, and ReAct under comparable experimental settings. See Weakness 2-2 in Reviewer ehTF, Weakness 1 in Reviewer VkDf, and Weakness 1 in Reviewer Xkgq.

  2. Ablation studies. We conducted ablation studies on structured tier records and the two reflection stages. See Weakness 2-2 in Reviewer ehTF.

  3. Benchmark details. We clarified the perturbation space, grid resolution, numerical tolerances, and number of valid solutions. See Weaknesses 1-2 and 1-3 in Reviewer ehTF.

  4. Evaluation seeds and table corrections. We corrected reporting errors in the original Table 2 and repeated the main experiments with three random seeds, reporting mean values and standard deviations. See Weakness 1-4 in Reviewer ehTF.

  5. Computational costs. We reported DFT and MLIP CPU-core hours, execution times, LLM token consumption, and API costs to evaluate search efficiency beyond DFT call counts. See Weakness 2-1 in Reviewer ehTF.

  6. Evolution under different training/test splits. We added ablation experiments to evaluate the effectiveness of evolution under two dataset splitting strategies: disjoint 1:1 training/test splits within each material-property combination, and splits that hold out entire materials or properties. See Weakness 2-2 in Reviewer ehTF.

  7. Transferability of learned experience. We evaluated the transfer of frozen search hints across materials and properties, including small-sample evolution and larger held-out benchmark task splits. See Weakness 2 in Reviewer VkDf.

  8. Cost of evolution. We measured the cost of learning search hints and compared it with the DFT savings obtained from reusing them on subsequent tasks. See Weakness 2 in Reviewer VkDf.

  9. Control for prior LLM knowledge. We conducted material-name masking and one-shot prediction experiments to examine whether the agent relies on memorized material knowledge. See Weakness 1-1 in Reviewer ehTF.

  10. Matbench Discovery analysis. We clarified the evaluation metrics and the interpretation of the SFBO results. We also introduced a recall-only baseline to examine potential memorization of public WBM data. See Weakness 2-3 in Reviewer ehTF.

  11. Related work and novelty. We discussed the distinctions between Struct-R1 and existing approaches, including MF-GFN, PhyVer, and LLEMA. See Weakness 1 in Reviewer Xkgq.

  12. Scope and generalizability. We clarified the empirical scope of our evaluation and acknowledged the need for further validation on larger search spaces. See Weakness 2 in Reviewer Xkgq.

Updates in the revised manuscript:

  1. Abstract and Introduction: We revised the descriptions of our contributions and empirical scope to avoid overstating the generality of Struct-R1.

  2. Related Work (Section 2.2): We expanded the discussion of related multi-fidelity and agentic materials methods, particularly MF-GFN and PhyVer, and clarified their differences from Struct-R1.

  3. Benchmark Design (Section 3.5): We added the grid resolution, numerical tolerances, and number of valid candidates for the benchmark tasks.

  4. Main Results (Table 2): We added four stronger baselines: LLMatDesign, LLEMA, MF-MES, and ReAct. We updated the results using three random seeds (0, 1, and 2) and reported mean values and standard deviations for overall performance. Some baseline results in the rebuttal were initially reported from seed 0, while the revised manuscript uses three-seed averages, resulting in slight numerical differences.

  5. Evolution Ablation and Transfer (Table 4): We added ablation studies of the two reflection stages using disjoint training and test tasks with GPT-5.6 Terra and GPT-5.2. We also evaluated the transferability of learned experience using splits that hold out entire materials or properties.

  6. Computational Cost (Table 5): We added a detailed comparison of DFT and MLIP CPU-core hours, execution time, LLM inference time, token consumption, and API costs.

  7. Matbench Discovery Analysis (Section 4.2): We clarified the performance comparison with MLIP ranking and added a recall-only control to examine the influence of prior LLM knowledge.

We appreciate the reviewers' time and consideration. We hope that these additional experiments and revisions address the major concerns raised during the review. We welcome further discussion and would be happy to clarify any remaining questions.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.