- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors introduce a benchmark for powder XRD for crystal structure tasks, a challenge for fast experimental characterization in materials science. The authors also introduce a model that is a strong baseline for the newly introduced pXRD benchmark.
Critical strengths
The benchmark introduced here seems valuable for the community (despite being incomplete, see below). The baseline model seems to be strong.
Critical weaknesses
Some widely studied tasks in this area (e.g. pXRD to space group, and also tasks without knowledge of the elements/composition) are lacking.
The paper discusses the sim-to-real challenge in this area, but the benchmark introduced here will not contribute to solving this. The benchmark only introduces new synthetic (simulated+augmented) data which can be generated cheaply (basically on the fly, faster than training, as demonstrated in the literature before), but this does not help the community to advance on bridging the sim-to-real gap. Testing on experimental data as requested by the benchmark is important, but training on a fixed synthetic dataset with better models (or more synthetic data, which is out of scope for a fixed benchmark) will not improve generalization to OOD experimental data.
The code/data availability (now or after publication) is not discussed in the manuscript. For code, the authors refer to the SI, where code is not discussed.
Bragglite seems to be a competitive model, but relevant related work and model details are not discussed in sufficient level of detail. The structure of the paper suggests that the augmentation is part of the model, which does not make sense to me, as it should rather be part of the benchmark. Why would you compare models with data augmentation to models without?
Questions for authors
Why did you leave out other tasks such as space group prediction?
Will you also release the code to generate more synthetic data, potentially also as part of the benchmark? Synthetic data is very cheap, so a fixed dataset does not make sense. Improvements of the approaches to modify the simulated diffratograms might help close the sim-to-real gap, but more of the same synthetic data will not close it because it follows a different distribution than the experimental data.
7 comments
W4-1. Insufficient discussion of related work and model details.
We thank the reviewer for this feedback. Prior PXRD-to-structure methods are discussed in Section 2, with implementation details provided in Appendix D.4. We will add the following extended comparison of model architectures, PXRD representations, and augmentation strategies to a dedicated appendix section:
Model architectures. Crystalyze uses a CDVAE-based latent diffusion model, while PXRDGen, XRDiff, and RealPXRD-Solver adopt CSPNet-based GNN backbones with diffusion- or flow-based generation. In contrast, Bragglite adopts the diffusion Transformer backbone introduced in Crystalite.
PXRD representations. Crystalyze and PXRDGen use dense PXRD profiles, whereas RealPXRD-Solver, XRDiff, and Bragglite adopt peak-based representations. While XRDiff also investigates full-profile inputs, its final model uses a peak-based representation. RealPXRD-Solver uses d-spacing and intensity as peak descriptors, while XRDiff additionally incorporates peak width and prominence. Bragglite adopts q-spacing, intensity, width, and prominence as peak descriptors.
PXRD augmentation. Prior methods employ different augmentation strategies, including peak broadening and background noise (Crystalyze), and peak shifts, intensity perturbations, weak-peak removal, and Gaussian noise (RealPXRD-Solver). XRDiff uses pseudo-Voigt profiles to model peak shapes when constructing its peak-based representation. Building on these prior approaches, Bragglite combines augmentation operations from existing methods and selects their configurations based on experimental validation.
W4-2. Unclear role of augmentation and fairness of baseline comparisons
We regard PXRD augmentation as a training-time strategy for improving sim-to-real transfer, which is why we describe it as part of Bragglite's training procedure in Section 4. Augmentation is applied on-the-fly during training to generate diverse augmented PXRD profiles, rather than being fixed as part of the benchmark dataset. Prior methods (e.g., Crystalyze and RealPXRD-Solver) also employ their own augmentation schemes. Thus, evaluating complete modeling pipelines with their respective augmentation strategies under a common benchmark protocol follows prior PXRD studies and enables fair pipeline-level comparisons. A similar practice is common in ImageNet benchmarks, where augmentation is treated as part of the training procedure (Cubuk et al., 2019; Cubuk et al., 2020). We will also release the augmentation pipeline alongside the benchmark code to facilitate its reuse and further development by the community for sim-to-real transfer.
To further isolate the contribution of augmentation and address concerns about baseline fairness, we additionally retrain RealPXRD-Solver and XRDiff using Bragglite's augmentation while retaining their respective model architectures. The results are summarized below.
| Method | MP-20 Top-1 (%) | RRUFF Top-1 (%) |
|---|---|---|
| RealPXRD-Solver (original) | 56.35 | 27.80 ± 4.76 |
| RealPXRD-Solver (+ Bragglite augmentation) | 56.51 | 35.50 ± 1.29 |
| XRDiff (original) | 55.10 | 17.80 ± 0.84 |
| XRDiff (+ Bragglite augmentation) | 54.41 | 33.25 ± 0.96 |
| Bragglite | 70.85 | 52.40 ± 0.89 |
Applying Bragglite's augmentation improves the experimental RRUFF performance of both baselines, while leaving their MP-20 performance largely unchanged. Nevertheless, Bragglite maintains a substantial performance advantage over both methods under the same augmentation setting, indicating that its improvements cannot be attributed to augmentation alone.
references:
[1] E. D. Cubuk et al., “AutoAugment: Learning Augmentation Strategies From Data,” CVPR, 2019. [2] E. D. Cubuk et al., “RandAugment: Practical Automated Data Augmentation with a Reduced Search Space,” CVPRW, 2020.
Thank you very much for replying to the questions.
Some of the replies are satisfying, but others are not.
The space group results are far below SOTA results despite the high tolerance factors, indicating that the model trained on a much more challenging task (crystal structure prediction) cannot solve a more simple tasks of space group prediction.
Providing the code is great and a mistake on my side for not seeing it. However, I still do not find it useful to phrase the paper as a "benchmark" with a fixed dataset. This task lives from access to practically infinite synthetic data on real or even synthetic crystal structures, with infinite possibilities for augmentation, so a fixed benchmark will not advance this community.
Furthermore, given the new results in response to reviewer PTKy regarding generalization to unseen prototypes (which is a usual split performed in papers addressing the aforementioned space group prediction task), which demonstrate that the performance substantially degrades for unseen prototypes, I will not raise my score. More analysis and improvement will be needed for the model to provide meaningful reliability in real-world tasks.
We thank the reviewer for recognizing the value of PXRD-Bench as a community resource and Bragglite as a strong baseline. We address each point below.
W1, Q1. Some widely studied tasks in this area (e.g., space group prediction, and also tasks without elements) are lacking.
Our benchmark focuses on full crystal structure determination from PXRD, aiming to recover the atomic-level structural information needed for downstream materials analysis. Space-group determination is also an important and standard step in PXRD-based structure analysis. Since Bragglite generates full crystal structures, their space groups can be directly analyzed. We report Top-1 and Top-20 accuracy on MP-20 under different symmetry tolerances (
angle_tolerance=5°).symprec(Å)The substantial improvement with increasing symmetry tolerance suggests that many generated structures closely preserve the reference symmetry, with small geometric deviations accounting for a considerable fraction of mismatches under stricter tolerances.
Regarding settings without elemental information, constituent elements are generally known from synthesis inputs or can be identified through complementary techniques such as energy-dispersive X-ray spectroscopy (EDS). We therefore focus on this practically motivated setting through our element-set task, which assumes known constituent elements but does not require exact primitive-cell composition.
W2. The benchmark introduced here will not contribute to solving sim-to-real challenge in this area.
PXRD-Bench facilitates sim-to-real research by combining a structure-paired simulated PXRD dataset with experimental PXRD validation and evaluation. Synthetic supervision enables models to learn crystallographic priors and PXRD-to-structure mappings, which can improve experimental performance with appropriate sim-to-real transfer methods. Figure 5b shows a strong positive correlation between simulated and experimental performance under our augmentation scheme. Moreover, our large-scale corpus enables systematic investigation of how sim-to-real transfer performance scales with synthetic training data. In a controlled scaling experiment within PXRD-Bench, increasing the training set size under a fixed augmentation scheme improves RRUFF performance (Figure 4).
Furthermore, experimental PXRD validation enables the development and selection of sim-to-real strategies, rather than optimizing simulated PXRD performance alone. For example, we select Bragglite trained with augmentation over its non-augmented counterpart despite its lower MP-20 performance, because it achieves better experimental validation performance. Separate experimental validation and test sets allow these strategies to be developed and evaluated without using test data for model selection.
W3, Q2. Code/data availability is not discussed in the manuscript.
The implementation code was provided as a separate ZIP archive in the OpenReview supplementary materials. We will explicitly point to this archive in the revised manuscript and clarify code and data availability. We have also updated the supplementary ZIP archive to include the complete end-to-end pipeline used to construct the datasets in our experiments.
The archive also includes the on-the-fly synthetic PXRD augmentation pipeline, allowing researchers to generate additional augmented patterns and explore alternative augmentation strategies.