- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a materials model based on large language models (LLMs) that can be used to design materials with specific properties or to predict material performance. The authors aim to construct a unified model for structures and text; the "producer-consumer" bridging mechanism they introduce is of significant practical value.
Critical strengths
- Unify sequence- and structure-based representations in materials science.
- Introduce an additional prompt-based benchmark for evaluation.
Critical weaknesses
see my question
Questions for authors
- I wonder whether T2C-FK performs better than a simple Best-of-N sampling strategy. As shown in Table 2. As shown in Table 4, ALM does not consistently outperform GPT-5.6-sol on OOD tasks. What might explain this relatively limited generalization performance?3. what the effect fo simple prompt for mattergen? 3. How would MatterGen perform with a simpler prompt-based conditioning approach? I believe this comparison is important to demonstrate whether the proposed LLM-based framework offers meaningful advantages over simpler conditional generation methods.
4 comments
Continuation of response to Reviewer hMvF's comments (#2)
Generalization relative to GPT-5.6 Sol
[Q2] As shown in Table 4, ALM does not consistently outperform GPT-5.6-sol on OOD tasks. What might explain this relatively limited generalization performance?
We first corrected an evaluation error. A bug had run Sol at the API's default (medium) reasoning effort instead of the stated low effort. Table 4 now reports Sol at correctly applied low effort (Table R13; both effort levels are in Table R12 of our response to Reviewer xtzu).
Table R13. Updated Table 4 (N=5×200, 95% CI over seeds).
| Model | Application ↑ | Describe (Comp.) ↑ | Describe (Struct.) ↑ | OOD (Comp.) ↑ | OOD (Struct.) ↑ |
|---|---|---|---|---|---|
| GPT-4.1 | 0.224±0.032 | 0.254±0.019 | 0.090±0.018 | 0.228±0.029 | 0.050±0.005 |
| GPT-5.2 | 0.252±0.051 | 0.356±0.014 | 0.162±0.013 | 0.366±0.026 | 0.108±0.008 |
| GPT-5.6 Sol (low) | 0.302±0.033 | 0.748±0.029 | 0.468±0.035 | 0.675±0.021 | 0.305±0.019 |
| ALM Edit | 0.323±0.020 | 0.730±0.031 | 0.412±0.041 | 0.641±0.032 | 0.316±0.022 |
To explain the remaining gap, we sorted every OOD prompt into one of five outcomes.
Table R14. Where OOD prompts fail (fraction of prompts, each row sums to 1).
| Model | Correct ↑ | Right composition, wrong structure | Wrong ratio of elements ↓ | Wrong elements ↓ | No structure ↓ |
|---|---|---|---|---|---|
| GPT-5.2 | 0.108 | 0.258 | 0.338 | 0.009 | 0.287 |
| GPT-5.6 Sol (low) | 0.305 | 0.370 | 0.255 | 0.003 | 0.067 |
| GPT-5.6 Sol (medium) | 0.357 | 0.339 | 0.264 | 0.003 | 0.037 |
| ALM Edit | 0.316 | 0.325 | 0.295 | 0.010 | 0.054 |
No model chooses the wrong elements more than about 1% of the time. ALM Edit's gap to Sol comes almost entirely from choosing the wrong stoichiometric ratio (0.295 vs 0.255). Because ALM Edit's language model writes the target composition before the diffusion model generates the structure, this reflects the underlying Qwen3-8B failing to recall the right formula rather than a failure of the bridge or the decoder. When ALM Edit gets the composition right, it finds the correct structure about as often as Sol.
To test this, we replaced only the composition-planning step with a newer, more knowledgeable language model (Qwen3.5-9B), leaving ALM Edit unchanged (Table R15).
Table R15. Replacing ALM Edit's composition planner (N=5×200, 95% CI over seeds).
| Model | Application ↑ | Describe (Comp.) ↑ | Describe (Struct.) ↑ | OOD (Comp.) ↑ | OOD (Struct.) ↑ |
|---|---|---|---|---|---|
| GPT-5.6 Sol (low) | 0.302±0.033 | 0.748±0.029 | 0.468±0.035 | 0.675±0.021 | 0.305±0.019 |
| GPT-5.6 Sol (medium) | 0.263±0.067 | 0.728±0.022 | 0.479±0.004 | 0.696±0.030 | 0.357±0.026 |
| ALM Edit | 0.323±0.020 | 0.730±0.031 | 0.412±0.041 | 0.641±0.032 | 0.316±0.022 |
| ALM Edit + Qwen3.5-9B planner | 0.512±0.067 | 0.864±0.019 | 0.329±0.016 | 0.780±0.026 | 0.371±0.030 |
With the stronger planner, ALM Edit beats both Sol settings on four of the five tasks, including OOD composition (0.780 vs 0.696) and structure (0.371 vs 0.357). The generalization gap therefore tracks the knowledge of the language model ALM is built on, and it shrinks as that model improves, without retraining the bridge or decoder.
Continuation of response to Reviewer hMvF's comments (#3)
MatterGen with prompt-based conditioning
[Q3] How would MatterGen perform with a simpler prompt-based conditioning approach? I believe this comparison is important to demonstrate whether the proposed LLM-based framework offers meaningful advantages over simpler conditional generation methods.
We agree that this comparison is central. We have added several additional baselines accordingly: a language model reads each prompt and writes a chemical system (e.g., "Li-Fe-P-O"), which conditions the released MatterGen [1] checkpoints, either with chemical_system alone or with chemical_system plus an energy-above-hull target of zero. MatterGen cannot take an exact count of each element or a structure to edit, so only the generation tasks are possible with this approach. We use two language models: Qwen3-8B, the same model ALM is built on, which isolates the effect of ALM's learned conditioning interface from the language model itself, and GPT-5.2, which has considerably broader materials knowledge.
Table R16. Prompt-conditioned MatterGen vs ALM Edit (N=5×200, 95% CI over seeds).
| Model | Application ↑ | Describe (Comp.) ↑ | Describe (Struct.) ↑ | OOD (Comp.) ↑ | OOD (Struct.) ↑ |
|---|---|---|---|---|---|
| Qwen3-8B → MatterGen (chemical system) | 0.344±0.045 | 0.080±0.015 | 0.029±0.006 | 0.058±0.010 | 0.020±0.003 |
| Qwen3-8B → MatterGen (chemical system + Ehull=0) | 0.360±0.028 | 0.092±0.020 | 0.028±0.013 | 0.056±0.014 | 0.019±0.007 |
| GPT-5.2 → MatterGen (chemical system) | 0.398±0.046 | 0.083±0.007 | 0.020±0.010 | 0.052±0.021 | 0.017±0.013 |
| GPT-5.2 → MatterGen (chemical system + Ehull=0) | 0.446±0.020 | 0.091±0.010 | 0.031±0.011 | 0.063±0.017 | 0.026±0.008 |
| ALM Edit | 0.323±0.020 | 0.730±0.031 | 0.412±0.041 | 0.641±0.032 | 0.316±0.022 |
With either composition planner (Qwen3-8B or GPT-5.2), the language model chooses the correct element set for at least 99.9% of Describe prompts, yet composition and structure recovery fall to 0.02 to 0.09, 7 to 14 times below ALM Edit. The language model is therefore not the bottleneck; MatterGen's published conditioning interface is. MatterGen's fields can name the elements but not their ratio, the structure type, or a requested change, so a request for SrTiO₃ becomes "some Sr-Ti-O compound". Prompt-conditioned MatterGen scores above ALM Edit only on Application consistency, where the judge mainly rewards a plausible choice of elements (0.360 vs 0.323 with Qwen3-8B, within CIs, and 0.446 with GPT-5.2). It also cannot perform any of the nine editing tasks, because MatterGen cannot take an input structure. ALM's producer-consumer bridge closes this gap by passing the language model's hidden states, which encode composition, structure, and the requested change, into every block of the diffusion model instead of reducing the request to a few condition labels.
We will add these sampling controls, the OOD error analysis, and the prompt-conditioned MatterGen comparison to the revised manuscript, and we hope they address the reviewer's questions and inspire a reconsideration of our score.
[1] Zeni, C. et al. A generative model for inorganic materials design. Nature 639, 624-632 (2025).
Thanks for the detailed responses and additional experiments. I appreciate the effort you've put into addressing my concerns, and I'll raise my score accordingly.
Response to Reviewer hMvF's comments (#1)
We thank the reviewer for recognizing the value of unifying language and structural representations for materials science and of a benchmark for prompt-based materials design. We agree that the benefits of T2C-FK and of learned language-to-structure conditioning should be tested against simpler alternatives, and that ALM's OOD performance needs a clearer explanation. To address these questions, we compared T2C-FK with best-of-64 sampling, analyzed OOD errors and tested an external composition planner, and evaluated language models coupled to the released conditional MatterGen checkpoints. We present the setup and results of each experiment below.
T2C-FK versus Best-of-N sampling
We compared T2C-FK with best-of-64 sampling using the same ALM Gen checkpoint and the same budget of 64 generation trajectories. T2C-FK steers 64 particles during denoising, while best-of-64 generates 64 unsteered samples and ranks them with the same stoichiometry reward. As shown in Table R9, T2C-FK raises MR@20 from 8.7% to 41.0% on MP-20 and from 2.8% to 10.0% on MPTS-52.
Table R9. ALM Gen CSP with and without T2C-FK (reproduced from our response to Reviewer xtzu; N=1000, K=64). Match rates in %.
T2C-FK is 3.7 times (MR@1) and 4.7 times (MR@20) better than best-of-64 on MP-20, and 3.0 times and 3.6 times better on MPTS-52, whose structures are up to 2.6 times larger. Best-of-N can only select among samples the model already produces. Without steering, just 1.4% of samples have the target stoichiometry, and 72% of compositions have none among 64. T2C-FK instead resamples particles toward the target during denoising, so it reaches compositions that the unsteered sampler almost never produces.