- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TRACE, a framework for LLM-agent-based multi-objective materials discovery that records evaluated parent–edit–child transitions and uses their observed property changes to guide subsequent local edits. Edits are ranked by their predicted ability to reduce the current candidate’s remaining constraint violations without damaging already satisfied objectives, while global LLM proposals maintain broader exploration. On 14 LLEMABench tasks with two quantized open-source backbones, TRACE improves hit rate over LLEMA on most tasks under approximately matched evaluation counts, supported by ablation, budget-allocation, and transition-level analyses.
Critical strengths
S1. Treating evaluated parent–edit–child transitions as feedback is intuitive and well motivated for multi-objective materials search.
S2. Under the same backbone, tasks, and oracle, with approximately matched property-evaluation counts, TRACE improves hit rate over LLEMA on most tasks.
S3. The empirical analysis covers multiple aspects of the method, including component ablations, exploration-budget allocation, transition-level diagnostics, and sensitivity to noisy property feedback.
S4. The figures are clear and informative. Figures 1 and 2 effectively illustrate the motivation and framework, while Figure 3 clearly presents discovery efficiency under a fixed attempt budget.
Critical weaknesses
W1. The controlled LLEMA–TRACE comparisons use only two 8-bit quantized backbones. Table 7 reports TRACE results with GPT-5.4 but no corresponding LLEMA results, leaving it unclear whether the relative gains persist with stronger proposers.
W2. Local edits preserve the parent lattice and coordinates, but it is unclear whether child structures are relaxed before evaluation. The paper also does not report how many hits come from reference data versus surrogate predictions, making their physical reliability difficult to assess.
W3. Although TRACE improves average performance under Mistral, some tasks show substantial regressions, such as Wide-Bandgap Semiconductors. Table 1 reports no standard deviations despite five runs, making the consistency of these differences difficult to assess.
m1. The ablation and budget-allocation studies are limited to Solid-State Electrolytes with a single backbone.
m2. The manuscript does not report the numerical weights in Eqs. 8, 12, and 14 or how they were selected.
Questions for authors
Q1. Could you report LLEMA results with GPT-5.4 or another stronger proposer, to show whether the relative gains of TRACE persist? (see W1)
Q2. Are child structures relaxed before property evaluation? What fraction of hits comes from reference data versus surrogate predictions? (see W2)
Q3. Could you report standard deviations for Table 1, and comment on why TRACE underperforms LLEMA on some tasks under Mistral? (see W3)
Q4. What values are used for the weights in Eqs. 8, 12, and 14, and how were they selected? (see m2)
5 comments
Response to Q2/W2:
We thank the reviewer for requesting this clarification. Following the common LLEMABench evaluation protocol, TRACE does not introduce an additional structure-relaxation stage after a local substitution. The edited CIF is evaluated through the same oracle pipeline used for LLEMA: properties are retrieved from the Materials Project when available, and unavailable properties are completed using the task-specific surrogate models. This shared protocol provides a controlled comparison of the two search methods without introducing method-specific evaluation differences and is stated explicitly in Appendix B.1 of the revised manuscript.
To make the source composition explicit, we audited valid hits across four representative tasks, two backbones, and five runs per setting. The table below reports the pooled percentage of valid hits across the five runs in two mutually exclusive categories. “Reference-assisted” means that the Materials Project supplied at least one constrained property and therefore includes mixed cases in which remaining properties were completed by surrogates. “Surrogate-only” means that no constrained property was obtained from the reference API.
| Backbone | Task | LLEMA Reference-assisted (%) | LLEMA Surrogate-only (%) | TRACE Reference-assisted (%) | TRACE Surrogate-only (%) |
|---|---|---|---|---|---|
| Qwen | Wide-Bandgap Semiconductors | 89.01 | 10.99 | 49.27 | 50.73 |
| Qwen | Solid-State Electrolytes | 75.00 | 25.00 | 65.66 | 34.34 |
| Qwen | High-k Dielectrics | 44.90 | 55.10 | 28.21 | 71.79 |
| Qwen | Piezo Energy Harvesters | 13.80 | 86.20 | 11.94 | 88.06 |
| Mistral | Wide-Bandgap Semiconductors | 93.47 | 6.53 | 92.83 | 7.17 |
| Mistral | Solid-State Electrolytes | 85.25 | 14.75 | 70.68 | 29.32 |
| Mistral | High-k Dielectrics | 45.57 | 54.43 | 36.96 | 63.04 |
| Mistral | Piezo Energy Harvesters | 12.99 | 87.01 | 10.61 | 89.39 |
TRACE has a lower reference-assisted fraction in all eight matched comparisons. Thus, in settings where TRACE improves, the gain cannot be explained by retrieving a larger fraction of known database materials. Our claims concern search performance under the shared LLEMABench oracle rather than independent experimental validation of the generated structures.
Response to Q3/W3:
We thank the reviewer for requesting task-level uncertainty estimates and a more explicit analysis of the task-dependent behavior under Mistral. We report the sample standard deviation over five independent runs for every task–backbone–method setting in Appendix D.1. The matched Mistral H.R. and Stab. results are reproduced below so that the corresponding gains and regressions can be assessed directly.
| Task | LLEMA H.R. (%) | TRACE H.R. (%) | LLEMA Stab. (%) | TRACE Stab. (%) |
|---|---|---|---|---|
| Wide-Bandgap Semiconductors | 51.98 ± 8.43 | 36.59 ± 5.68 | 43.67 ± 7.58 | 29.50 ± 3.71 |
| SAW/BAW Acoustic Substrates | 27.54 ± 6.77 | 43.21 ± 6.62 | 6.65 ± 2.70 | 14.70 ± 9.31 |
| High-k Dielectrics | 11.09 ± 2.64 | 12.38 ± 4.14 | 5.28 ± 1.80 | 4.85 ± 1.70 |
| Solid-State Electrolytes | 31.96 ± 6.21 | 60.06 ± 16.98 | 27.35 ± 3.91 | 48.15 ± 11.33 |
| Piezo Energy Harvesters | 69.98 ± 4.34 | 66.23 ± 7.63 | 7.77 ± 3.00 | 15.53 ± 2.53 |
| Transparent Conductors | 21.62 ± 3.31 | 27.65 ± 6.99 | 10.01 ± 1.89 | 11.26 ± 3.68 |
| Insulating Dielectrics | 15.32 ± 2.63 | 16.62 ± 5.24 | 8.75 ± 3.32 | 9.59 ± 4.65 |
| Photovoltaic Absorbers | 21.78 ± 4.11 | 22.96 ± 4.54 | 13.52 ± 3.40 | 8.96 ± 1.80 |
| Hard Coating Materials | 9.91 ± 4.79 | 7.28 ± 3.28 | 2.92 ± 1.44 | 2.17 ± 1.25 |
| Hard, Stiff Ceramics | 52.32 ± 14.53 | 57.42 ± 13.15 | 19.87 ± 5.69 | 11.59 ± 3.14 |
| Aerospace Materials | 9.05 ± 2.44 | 5.92 ± 1.64 | 2.11 ± 0.97 | 1.13 ± 1.29 |
| Acousto-optic Hybrids | 11.67 ± 2.79 | 14.28 ± 3.41 | 4.36 ± 1.79 | 8.32 ± 6.61 |
| Low-Density Structures | 16.63 ± 4.84 | 19.07 ± 7.37 | 2.14 ± 1.59 | 3.30 ± 1.14 |
| Perovskite Oxides | 4.60 ± 1.28 | 12.70 ± 4.75 | 3.68 ± 0.98 | 2.45 ± 1.55 |
Under Mistral, TRACE increases the macro-average H.R. from 25.39% to 28.74% and has a higher mean H.R. on 10 of the 14 tasks. The run-level analysis therefore supports a majority-task and macro-average improvement while showing that the effect magnitude and variability remain task dependent.
To address the second part of the question, we use the route-level audit in Appendix D.4 to examine two representative regressions: the largest H.R. decrease on Wide-Bandgap Semiconductors and a distinct route-balance pattern on Piezo Energy Harvesters. On Wide-Bandgap Semiconductors, H.R. decreases from 51.98 ± 8.43% for LLEMA to 36.59 ± 5.68% for TRACE. The regression cannot be attributed to TRACE evaluating fewer candidates: the corresponding means are 156.0 ± 5.1 and 158.0 ± 1.0. The route-level audit provides a concrete diagnostic: under Mistral, post-initialization valid-candidate yield is 55.20% for LLEMA, compared with 41.52% for TRACE's global route and 43.01% for its local route. The direction reverses with Qwen, where the corresponding yields are 25.14%, 34.24%, and 60.29%, and H.R. increases from 25.23% to 44.78%. This matched reversal indicates that the regression is specific to the backbone–task combination rather than a general failure on Wide-Bandgap Semiconductors.
Piezo Energy Harvesters illustrates a different route-balance effect. Under Mistral, TRACE's local route has a higher valid-candidate yield than LLEMA (79.25% versus 69.65%), whereas its global route is weaker (57.67%). This route imbalance is consistent with a slightly lower aggregate H.R. (66.23 ± 7.63% versus 69.98 ± 4.34%), while Stab. increases from 7.77 ± 3.00% to 15.53 ± 2.53%. Together, these results indicate that TRACE's gains depend jointly on the backbone-conditioned proposal distribution and the effectiveness and balance of its global and local routes. We therefore characterize the Mistral result as a majority-task and macro-average improvement with identifiable task-specific regressions.
Response to Q1/W1:
We thank the reviewer for requesting a matched comparison with a stronger proposer. To provide the requested controlled comparison, we evaluated both LLEMA and TRACE with GPT-5.6 on four matched tasks under the same task definitions, oracle pipeline, and budget settings used in the main comparison. Each entry reports one controlled run.
TRACE improves both H.R. and Stab. on Wide-Bandgap Semiconductors and Solid-State Electrolytes. On SAW/BAW Acoustic Substrates, H.R. remains nearly unchanged while Stab. increases from 5.03% to 20.25%; on Acousto-optic Hybrids, H.R. increases from 16.22% to 27.92%, while Stab. changes from 4.05% to 3.25%. This task-level comparison provides evidence that TRACE's relative gains are not confined to the two quantized backbones, while the affected metric and gain magnitude remain task dependent. The comparison is included in Appendix D.7 of the revised manuscript.