acceptodds
Under review as a conference paper at ICLR 2027

Generating Materials with Excellent Properties via Actor-Critic Diffusion

Abstract

Diffusion models, as the basis for state-of-the-art foundation models for materials discovery, potentially signal a paradigm shift from the expensive, manual trial-and-error industrial practice. Technology across multiple fields is powered by materials with desirable and often rarely seen properties. Classifier-free (or guided) diffusion has been the main approach in generating materials with these conditioned properties, however, these approaches tend to fall back to existing materials when conditioned on extreme property values. This poses significant hurdles in discovering novel, stable materials with excellent properties. Instead of formulating the problem as conditional diffusion, we define a novel Reinforcement Learning (RL) process for the reverse diffusion to maximize property values as rewards. RL explores the composition space freely, hence not bounded by existing materials in the training dataset. We present CrystRL, a reinforcement learning-based method that can generate novel, stable materials with excellent properties. CrystRL fine-tunes the unconditional reverse diffusion using an actor network, which suggests denoising moves to good materials. It is supplemented by a critic network, which predicts the best property values from a noisy state. Experimental results show that CrystRL discovers novel and stable crystals with out-of-training-distribution ML-predicted bulk moduli of up to 408 GPa and DFT-verified bulk moduli of up to 397 GPa more consistently than classifier-free guided generation.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI for scientific discovery, Reinforcement learning, Diffusion, Materials discovery, Actor-critic reinforcement learning, Guided diffusion, Conditional diffusion, Superhard materials

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_o9b8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CrystRL, an RL-based fine-tuning procedure for pretrained crystal diffusion models, demonstrated on MatterGen. Instead of conditioning generation on a target property value, the reverse denoising process is formulated as an MDP and fine-tuned with an actor–critic algorithm so as to maximize the property of the relaxed final structure — bulk modulus throughout the experiments. The method handles the hybrid action space by treating continuous variables (coordinates, lattice) and discrete variables (atomic species) as separate policy factors with their own clipped surrogate losses, stabilizes training by freezing the backbone and updating only ~133k LoRA parameters on the output projections, and trains a time-conditioned critic, initialized from the score network, to estimate the terminal reward from intermediate noisy states. On Alex-MP-20, CrystRL raises the yield of high-modulus candidates relative to MatterGen conditioned at 320 GPa while keeping the usual stability/uniqueness/novelty (S.U.N.) metrics within a reasonable range; the top C–Re candidates were additionally checked with VASP.

Critical strengths

1.The problem being attacked — saturation of guided diffusion at extreme conditioning values — is a genuine bottleneck for property-targeted materials generation, and Figure 1 demonstrates it convincingly with the official checkpoint. 2.The hybrid update scheme is well matched to crystallographic data. Separating continuous coordinate updates from discrete species updates with distinct clipping thresholds (ϵc\epsilon_c versus ϵd\epsilon_d) reflects the actual geometry of the two spaces rather than forcing one framework onto both. 3.Freezing the backbone and training a small set of LoRA parameters is a pragmatic and, judging by the training curves, effective guard against policy collapse. Initializing the time-conditioned critic from the score network is economical.

Critical weaknesses

1.Reward-metric circularity, with DFT verification on six hand-picked structures only. The observed 10–15 GPa ML–DFT offset on those six is small enough to accept at the top, but the bias across the rest of the range is unknown. 2.No selection baseline. Training consumed roughly twenty thousand rollouts, each scored by a MatterSim relaxation. Under the same budget, sampling from the baseline and keeping the best is the natural null, and since MatterGen's best (405.09 GPa) already sits within ~3 GPa of CrystRL's best (408.21 GPa) it is not obvious that the RL run wins on the maximum. The comparison should be made, and the baseline should also be shown at conditioning values beyond 320 GPa. 3.All ≥400 GPa structures are C–Re binaries, and three of the four highlighted compositions (C$_2$Re$_4$, C$_3$Re$_6$, C$_3$Re$_9$) are at Re:C = 2:1, i.e. the Re$_2$C stoichiometry, a composition with a substantial prior literature as an ultraincompressible carbide. Novelty assessed by structure matching against Alexandria+MP-20 cannot see that literature, nor ICSD or OQMD. The generated phases should be compared against known and predicted Re–C phases before any novelty claim is made. 4.No ablations (critic on/off, clipping thresholds, LoRA rank, KL weight, entropy coefficient — the last is not even reported), and everything comes from a single training run with no error bars. The ≥400 GPa claim is 6 out of 512. 5.The critic's error grows to ~62 GPa MAE at full noise (Appendix B), and the states the policy visits as training progresses are out of distribution for the critic by construction. The possibility that the actor exploits this is acknowledged implicitly but never probed. 6.The generated structures require post-hoc relaxation before their properties are realized; the model produces pre-relaxed states rather than local minima natively. The authors are upfront about this in the limitations section.

Questions for authors

1.Did you observe reward hacking during training — structures that MatterSim scores highly but that are unphysical or clearly non-synthesizable when inspected? 2.How sensitive are the results to the asymmetric clipping (ϵc=0.2\epsilon_c = 0.2 versus ϵd=0.1\epsilon_d = 0.1)? What happens with equal thresholds or a single unified ratio? 3.Optimizing bulk modulus alone pushes compositions toward heavy, dense elements; the C–Re concentration is predictable in hindsight. How easily could a hull-distance or synthesizability term be folded into the reward without destabilizing PPO? 4.Where exactly does the 405.09 GPa MatterGen structure come from (conditioning value, sample count, compute), and how should it be reconciled with the 0.00% entry in Table 1? 5.Can you provide DFT moduli for a random (not top-scored) subsample of generated structures, and EhullE_{\mathrm{hull}} values for the Appendix D structures against all competing C–Re phases? 6.Under a matched relaxation budget, how does best-of-NN selection from the baseline compare? 7.Is the fine-tuned policy conditional or unconditional during rollouts? The abstract says the unconditional reverse diffusion is fine-tuned, but Section 4 initializes from a model equipped with property-conditioning adapters.

Reviewer_MfL3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes CrystRL, a method to generate novel, stable crystals with extreme property values by fine-tuning the pretrained MatterGen diffusion model with actor-critic reinforcement learning (PPO-style updates through LoRA adapters on the frozen backbone). The authors argue that classifier-free guidance cannot reach extreme property values: when conditioned on them, the generated property saturates and samples fall back to known materials. They therefore replace conditioning with RL fine-tuning of the unconditional model. The reverse diffusion is cast as a Markov decision process (MDP) whose action is the denoising step at=xt−1−xta_t = x_{t-1} - x_t. The actor takes continuous actions (fractional atomic coordinates and lattice parameters) and discrete actions (chemical species). In order to avoid costly DFT simulations, the final reward r(x0)r(x_0) is computed using a machine-learned force field (MatterSim), and a surrogate critic estimates the value function at each step. All models are pretrained on Alex-MP-20, and the task is generating crystal structures with high bulk modulus. The only baseline is MatterGen with classifier-free guidance conditioned at 320 GPa. Results show that CrystRL generates crystals with higher bulk modulus than the MatterGen baseline, including a few samples with ML-predicted values above 400 GPa, which are not present in the training set.

Critical strengths
  • The paper addresses a relevant limitation of classifier-free guidance: it struggles to generate samples with property values outside the training distribution. The authors respond with a simple and seemingly effective idea, fine-tuning the diffusion model with RL to maximize the target property directly.
  • CrystRL shifts the generated bulk modulus distribution upward compared with the MatterGen baseline (52.5% vs 20.1% of samples above 350 GPa) while keeping a comparable stability rate.
Critical weaknesses
  • Although the idea is sensible, its novelty is limited: the method mainly combines existing components (DDPO-style RL fine-tuning, an actor-critic objective, and LoRA adaptation of MatterGen) to address what is essentially an engineering problem.
  • Relevant references are missing, such as Park & Walsh (Nature Machine Intelligence, 2026), which addresses a similar problem by fine-tuning a crystal diffusion model with RL as an alternative to classifier-free guidance, or Xu et al. (Nature Communications, 2026) and DPOK (Fan et al., NeurIPS 2023), which combine diffusion and RL to steer generation, the latter already learning a value function on intermediate noisy states.
  • The proposed method is compared against a single baseline, MatterGen with classifier-free guidance.
  • The evaluation relies on a single dataset and a single target property, and reports results from one run without standard deviations or confidence intervals, which makes it difficult to assess robustness.
  • The baseline is conditioned at 320 GPa but evaluated at thresholds of 350–400 GPa. Although the authors justify this choice by diminishing returns and a loss of novelty at higher conditioning values, Figure 1 suggests that the mean bulk modulus keeps increasing, with MatterGen conditioned at 360–480 GPa achieving higher mean bulk moduli (≈355–378 GPa) than CrystRL (343.9 GPa). A systematic comparison across conditioning values, reporting both bulk modulus and S.U.N. metrics, is needed.
  • The claim in the introduction that "the guidance mechanism safely defaults to […] the nearest known stable material" is stated without supporting evidence or reference.

References

Questions for authors
  1. How does MatterGen fare when conditioned on higher values (e.g., 400–480 GPa), in terms of the Table 1 thresholds and S.U.N. rate?
  2. Have the authors compared with other RL fine-tuning approaches, e.g., GRPO-style updates as in Park & Walsh (2026)?
  3. Did the authors conduct an ablation study without the critic, using only the terminal reward for policy optimization, as in standard DDPO?
  4. In Table 1, bulk modulus is evaluated on 512 samples from the final CrystRL checkpoint, whereas in Table 2, stability, uniqueness and novelty are averaged cumulatively over training, while the baseline appears to be evaluated on a single batch. Is there a reason for this difference? Could the authors report S.U.N. metrics for the final checkpoint on the same 512 samples?
  5. In Section 3.2, the advantage is computed as r(x0)−V(xt,t)r(x_0) - V(x_t, t), whereas Algorithm 1 uses the difference between two consecutive critic estimates, V(xt−1,t−1)−V(xt,t)V(x_{t-1}, t-1) - V(x_t, t). Which formula was actually implemented?
Reviewer_g3QrOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CrystRL, which reframes the reverse diffusion process of MatterGen as a reinforcement-learning problem for discovering crystals with extreme out-of-distribution properties. The frozen MatterGen backbone is adapted with LoRA actor heads for continuous geometry and discrete atomic species, while a time-conditioned GemNet critic predicts terminal bulk modulus from noisy intermediate states and supplies denoising-step guidance. The policy is optimized with a PPO/DDPO-style objective, separate clipping for continuous and discrete actions, a discrete KL anchor, and MatterSim-derived terminal rewards. On Alex-MP-20, a final 512-sample generation pass shifts the relaxed bulk-modulus distribution above MatterGen conditioned at 320 GPa: 52.54% versus 20.12% of structures exceed 350 GPa, and 1.18% exceed 400 GPa under the ML evaluator. Cumulative training samples also show a slightly higher stable-unique-novel fraction (24.18% versus 20.71%). Four Re-rich carbon candidates are independently evaluated with DFT and have reported bulk moduli of 386.70–397.03 GPa.

Critical strengths
  1. The paper presents a technically interesting and well-motivated way to steer a mixed continuous/discrete crystal diffusion model beyond ordinary conditional interpolation. The combination of an actor-critic formulation, modality-specific PPO clipping, a KL anchor, and low-rank adaptation is coherent and potentially reusable for other materials objectives.

  2. The evaluation goes beyond reporting only surrogate rewards: the authors measure relaxed bulk modulus, stability, uniqueness, and novelty, analyze critic error across noise levels, and provide first-principles calculations for selected top candidates. The sizable shift in the full 512-sample bulk-modulus distribution is more informative than the small difference between individual maxima.

Critical weaknesses
  1. The principal comparison is not compute- or search-budget matched. Table 1 compares 512 samples from the final CrystRL checkpoint with 512 samples from a single MatterGen setting conditioned at 320 GPa, but CrystRL first consumes hundreds of RL epochs and many MatterSim evaluations. The paper does not compare against a tuned Pareto frontier over conditioning values/guidance strengths or other optimization baselines, and it provides no ablations for the critic, LoRA, separate clipping, KL term, or timestep subsampling. Consequently, the reported gain cannot yet be attributed to the proposed actor-critic design rather than additional optimization budget or particular implementation choices.

  2. Statistical and physical validation are too limited for the strongest claims. Results appear to come from one main training run with no seed-level uncertainty, and MatterSim is used both to generate the reward and to evaluate most outcomes, creating a risk of evaluator over-optimization. DFT is reported only for four selected C–Re candidates. Negative formation energy relative to elements does not by itself establish convex-hull, dynamical, or mechanical stability, and the 408.21 versus 405.09 GPa maximum is explicitly within surrogate error. The concentration of all >400 GPa samples in a narrow Re-rich family also raises a mode-concentration concern.

Questions for authors
  1. Under a matched total budget that includes RL rollouts and MatterSim evaluations, how does CrystRL compare with MatterGen tuned over conditioning targets and guidance scales, and with simpler reward-guided search or policy-gradient baselines? Please include multiple training seeds and ablations of the critic, LoRA restriction, separate clipping, KL anchor, and timestep subsampling.

  2. Can the authors validate a broader, pre-specified set of candidates (for example, all >400 GPa candidates plus a random or stratified sample) using DFT relaxation and DFT energy-above-hull, elastic-tensor, and phonon calculations? How many distinct compositions/prototypes remain after deduplication, and how sensitive are the conclusions to excluding Re-rich structures?

  3. Please reconcile the advantage definitions and critic targets. Section 3.2 states A_t = r(x_0) - V(x_t,t) and calls this equivalent to GAE with gamma=lambda=1, whereas Algorithm 1 uses V(x_{t-1},t-1) - V(x_t,t) for t>1 and bootstrapped critic targets, while Appendix E.4 trains the critic directly against terminal reward. Which estimator and target are actually used in the experiments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.