acceptodds
Under review as a conference paper at ICLR 2027

PXRD-Bench: How Close Are We to Determining Crystal Structures from Powder Diffraction?

Abstract

Determining crystal structures from powder X-ray diffraction (PXRD) would enable routine characterization of newly synthesized materials, including those absent from reference databases. Generative models offer a promising approach, but existing evaluations remain fragmented and largely assume single-phase samples with known composition. To fill this gap, we introduce PXRD-Bench, a unified benchmark that extends evaluation to unknown per-element atom counts and recovery of both structures in two-phase mixtures. It provides approximately 4.1 million training structures with simulated PXRD, tests generalization to unseen compositions and structural prototypes, and evaluates recovery from experimental PXRD after excluding structurally matching training entries. To provide a strong reference baseline, we also introduce Bragglite, a PXRD-to-structure model developed through a systematic study of PXRD modeling choices. It achieves a 44.2% Top-1 match rate on experimental RRUFF patterns with known composition, 24.6% above the strongest prior baseline. Further analysis yields several key findings: some models fail to benefit from experimental PXRD, while generalization to unseen structural prototypes and recovery from two-phase mixtures remain challenging. Together, our benchmark, systematic studies, and Bragglite clarify the capabilities and limitations of current PXRD-to-structure models.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Powder X-ray diffraction (PXRD), Crystal structure determination, Sim-to-real transfer, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UjrxOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce a benchmark for powder XRD for crystal structure tasks, a challenge for fast experimental characterization in materials science. The authors also introduce a model that is a strong baseline for the newly introduced pXRD benchmark.

Critical strengths

The benchmark introduced here seems valuable for the community (despite being incomplete, see below). The baseline model seems to be strong.

Critical weaknesses

Some widely studied tasks in this area (e.g. pXRD to space group, and also tasks without knowledge of the elements/composition) are lacking.

The paper discusses the sim-to-real challenge in this area, but the benchmark introduced here will not contribute to solving this. The benchmark only introduces new synthetic (simulated+augmented) data which can be generated cheaply (basically on the fly, faster than training, as demonstrated in the literature before), but this does not help the community to advance on bridging the sim-to-real gap. Testing on experimental data as requested by the benchmark is important, but training on a fixed synthetic dataset with better models (or more synthetic data, which is out of scope for a fixed benchmark) will not improve generalization to OOD experimental data.

The code/data availability (now or after publication) is not discussed in the manuscript. For code, the authors refer to the SI, where code is not discussed.

Bragglite seems to be a competitive model, but relevant related work and model details are not discussed in sufficient level of detail. The structure of the paper suggests that the augmentation is part of the model, which does not make sense to me, as it should rather be part of the benchmark. Why would you compare models with data augmentation to models without?

Questions for authors

Why did you leave out other tasks such as space group prediction?

Will you also release the code to generate more synthetic data, potentially also as part of the benchmark? Synthetic data is very cheap, so a fixed dataset does not make sense. Improvements of the approaches to modify the simulated diffratograms might help close the sim-to-real gap, but more of the same synthetic data will not close it because it follows a different distribution than the experimental data.

7 comments

We thank the reviewer for recognizing the value of PXRD-Bench as a community resource and Bragglite as a strong baseline. We address each point below.


W1, Q1. Some widely studied tasks in this area (e.g., space group prediction, and also tasks without elements) are lacking.

Our benchmark focuses on full crystal structure determination from PXRD, aiming to recover the atomic-level structural information needed for downstream materials analysis. Space-group determination is also an important and standard step in PXRD-based structure analysis. Since Bragglite generates full crystal structures, their space groups can be directly analyzed. We report Top-1 and Top-20 accuracy on MP-20 under different symmetry tolerances (angle_tolerance=5°).

symprec (Å)Stoichiometry Top-1Stoichiometry Top-20Element-set + N Top-1Element-set + N Top-20
0.152.48%71.58%23.92%44.11%
0.268.42%81.31%40.29%63.04%
0.371.64%82.73%46.78%69.47%

The substantial improvement with increasing symmetry tolerance suggests that many generated structures closely preserve the reference symmetry, with small geometric deviations accounting for a considerable fraction of mismatches under stricter tolerances.

Regarding settings without elemental information, constituent elements are generally known from synthesis inputs or can be identified through complementary techniques such as energy-dispersive X-ray spectroscopy (EDS). We therefore focus on this practically motivated setting through our element-set task, which assumes known constituent elements but does not require exact primitive-cell composition.


W2. The benchmark introduced here will not contribute to solving sim-to-real challenge in this area.

PXRD-Bench facilitates sim-to-real research by combining a structure-paired simulated PXRD dataset with experimental PXRD validation and evaluation. Synthetic supervision enables models to learn crystallographic priors and PXRD-to-structure mappings, which can improve experimental performance with appropriate sim-to-real transfer methods. Figure 5b shows a strong positive correlation between simulated and experimental performance under our augmentation scheme. Moreover, our large-scale corpus enables systematic investigation of how sim-to-real transfer performance scales with synthetic training data. In a controlled scaling experiment within PXRD-Bench, increasing the training set size under a fixed augmentation scheme improves RRUFF performance (Figure 4).

Furthermore, experimental PXRD validation enables the development and selection of sim-to-real strategies, rather than optimizing simulated PXRD performance alone. For example, we select Bragglite trained with augmentation over its non-augmented counterpart despite its lower MP-20 performance, because it achieves better experimental validation performance. Separate experimental validation and test sets allow these strategies to be developed and evaluated without using test data for model selection.


W3, Q2. Code/data availability is not discussed in the manuscript.

The implementation code was provided as a separate ZIP archive in the OpenReview supplementary materials. We will explicitly point to this archive in the revised manuscript and clarify code and data availability. We have also updated the supplementary ZIP archive to include the complete end-to-end pipeline used to construct the datasets in our experiments.

The archive also includes the on-the-fly synthetic PXRD augmentation pipeline, allowing researchers to generate additional augmented patterns and explore alternative augmentation strategies.

W4-1. Insufficient discussion of related work and model details.

We thank the reviewer for this feedback. Prior PXRD-to-structure methods are discussed in Section 2, with implementation details provided in Appendix D.4. We will add the following extended comparison of model architectures, PXRD representations, and augmentation strategies to a dedicated appendix section:

Model architectures. Crystalyze uses a CDVAE-based latent diffusion model, while PXRDGen, XRDiff, and RealPXRD-Solver adopt CSPNet-based GNN backbones with diffusion- or flow-based generation. In contrast, Bragglite adopts the diffusion Transformer backbone introduced in Crystalite.

PXRD representations. Crystalyze and PXRDGen use dense PXRD profiles, whereas RealPXRD-Solver, XRDiff, and Bragglite adopt peak-based representations. While XRDiff also investigates full-profile inputs, its final model uses a peak-based representation. RealPXRD-Solver uses d-spacing and intensity as peak descriptors, while XRDiff additionally incorporates peak width and prominence. Bragglite adopts q-spacing, intensity, width, and prominence as peak descriptors.

PXRD augmentation. Prior methods employ different augmentation strategies, including peak broadening and background noise (Crystalyze), and peak shifts, intensity perturbations, weak-peak removal, and Gaussian noise (RealPXRD-Solver). XRDiff uses pseudo-Voigt profiles to model peak shapes when constructing its peak-based representation. Building on these prior approaches, Bragglite combines augmentation operations from existing methods and selects their configurations based on experimental validation.


W4-2. Unclear role of augmentation and fairness of baseline comparisons

We regard PXRD augmentation as a training-time strategy for improving sim-to-real transfer, which is why we describe it as part of Bragglite's training procedure in Section 4. Augmentation is applied on-the-fly during training to generate diverse augmented PXRD profiles, rather than being fixed as part of the benchmark dataset. Prior methods (e.g., Crystalyze and RealPXRD-Solver) also employ their own augmentation schemes. Thus, evaluating complete modeling pipelines with their respective augmentation strategies under a common benchmark protocol follows prior PXRD studies and enables fair pipeline-level comparisons. A similar practice is common in ImageNet benchmarks, where augmentation is treated as part of the training procedure (Cubuk et al., 2019; Cubuk et al., 2020). We will also release the augmentation pipeline alongside the benchmark code to facilitate its reuse and further development by the community for sim-to-real transfer.

To further isolate the contribution of augmentation and address concerns about baseline fairness, we additionally retrain RealPXRD-Solver and XRDiff using Bragglite's augmentation while retaining their respective model architectures. The results are summarized below.

MethodMP-20 Top-1 (%)RRUFF Top-1 (%)
RealPXRD-Solver (original)56.3527.80 ± 4.76
RealPXRD-Solver (+ Bragglite augmentation)56.5135.50 ± 1.29
XRDiff (original)55.1017.80 ± 0.84
XRDiff (+ Bragglite augmentation)54.4133.25 ± 0.96
Bragglite70.8552.40 ± 0.89

Applying Bragglite's augmentation improves the experimental RRUFF performance of both baselines, while leaving their MP-20 performance largely unchanged. Nevertheless, Bragglite maintains a substantial performance advantage over both methods under the same augmentation setting, indicating that its improvements cannot be attributed to augmentation alone.

references:

[1] E. D. Cubuk et al., “AutoAugment: Learning Augmentation Strategies From Data,” CVPR, 2019. [2] E. D. Cubuk et al., “RandAugment: Practical Automated Data Augmentation with a Reduced Search Space,” CVPRW, 2020.

Thank you very much for replying to the questions.

Some of the replies are satisfying, but others are not.

The space group results are far below SOTA results despite the high tolerance factors, indicating that the model trained on a much more challenging task (crystal structure prediction) cannot solve a more simple tasks of space group prediction.

Providing the code is great and a mistake on my side for not seeing it. However, I still do not find it useful to phrase the paper as a "benchmark" with a fixed dataset. This task lives from access to practically infinite synthetic data on real or even synthetic crystal structures, with infinite possibilities for augmentation, so a fixed benchmark will not advance this community.

Furthermore, given the new results in response to reviewer PTKy regarding generalization to unseen prototypes (which is a usual split performed in papers addressing the aforementioned space group prediction task), which demonstrate that the performance substantially degrades for unseen prototypes, I will not raise my score. More analysis and improvement will be needed for the model to provide meaningful reliability in real-world tasks.

Reviewer_PTKyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PXRD-Bench, a benchmark for crystal structure determination from powder X-ray diffraction (PXRD). It includes a large-scale simulated dataset, several OOD evaluation settings, and tasks involving unknown stoichiometry and two-phase mixtures. The authors also develop Bragglite, a PXRD-conditioned crystal diffusion model, and evaluate it against existing methods. The results show substantial improvements in structure recovery on experimental PXRD and reveal that generalization to unseen structural prototypes remains challenging.

Critical strengths
  1. The problem is important, and a more systematic benchmark for PXRD-based structure determination would be useful to the community.

  2. The benchmark covers several meaningful settings, especially prototype-level OOD generalization and two-phase recovery. The large-scale dataset and leakage-controlled splits are also valuable.

  3. Bragglite achieves strong results on both simulated and experimental PXRD. The analysis of PXRD representations, data augmentation, and sim-to-real transfer provides some useful insights.

Critical weaknesses
  1. I have concerns about the fairness of the baseline comparisons. Bragglite and the existing methods are trained with substantially different optimization settings and training budgets. For example, Bragglite is trained for 800K steps on MP-20, while the baselines are trained for at most 300K steps. Bragglite also benefits from a carefully optimized PXRD augmentation pipeline, which is not applied to the other methods. Since this is a benchmark paper, I think fair comparison is particularly important. The current results do not clearly separate the improvements from model architecture, training budget, and augmentation strategy. At least, the authors should examine whether the baselines also benefit from the proposed training recipe and whether they are sufficiently converged.

  2. The prototype-OOD results need more systematic analysis. One of the main findings is that performance drops significantly on unseen structural prototypes, while remaining high on unseen compositions. However, these splits may differ in other important aspects, such as atom count, crystal symmetry, and structural complexity. It is therefore unclear how much of the performance gap is actually caused by unseen prototypes rather than differences in the difficulty of the test samples. I think more controlled or stratified experiments are needed to support this conclusion.

  3. Some dataset construction choices may introduce selection bias. In Appendix A.1, the authors discard entire composition buckets containing more than 128 records before deduplication. This seems like a rather aggressive filtering strategy. In particular, it may remove compositions with many polymorphs, which could be among the more challenging cases for PXRD-based structure determination. I would like to understand how many structures and compositions are removed by this rule, and whether it affects the difficulty and representativeness of the resulting benchmark.

  4. The two-phase recovery task still relies on strong oracle information. The model is given the true primitive-cell atom counts of both phases, which are not generally available from a mixed experimental PXRD pattern. This simplifies an important part of the actual structure determination problem. I think the task is still meaningful as a controlled evaluation, but its practical implications should be stated more carefully. An ablation using predicted rather than oracle atom counts, or an analysis of the impact of this information, would make the contribution more convincing.

  5. The experimental evaluation is relatively limited, especially for the sim-to-real conclusions. The RRUFF and opXRD test sets contain only 100 and 83 samples, and the experimental two-phase evaluation uses just 21 mixtures. Meanwhile, several representation and augmentation choices are selected using experimental validation sets with only 48 and 40 samples. This raises concerns about how well the findings generalize to other experimental distributions. The reported variation across sampling seeds does not fully address this issue. I think stronger evidence of robustness across experimental datasets or conditions would be useful.

Questions for authors
  1. Can the authors provide more controlled baseline comparisons, especially regarding training budgets and PXRD augmentation? How much of Bragglite's advantage remains if the same augmentation pipeline is applied to the existing methods?

  2. Can the authors provide a more detailed analysis of prototype-OOD performance, controlling for atom count, crystal system, and other structural complexity factors? This would help establish whether prototype novelty is indeed the main reason for the large performance drop.

  3. What is the motivation for removing entire composition buckets with more than 128 records? How many structures and compositions are excluded, and could this filtering remove polymorph-rich or otherwise difficult cases?

  4. How much does the two-phase recovery performance depend on the oracle per-phase atom counts? Can the authors evaluate a setting without this information, or at least quantify its contribution?

  5. Given the relatively small experimental validation and test sets, can the authors provide further evidence that the chosen PXRD representations and augmentation strategies generalize beyond the specific experimental datasets used here?

6 comments

We thank the reviewer for recognizing the value of our large-scale, leakage-controlled benchmark, diverse evaluation settings, and Bragglite's strong results and insightful sim-to-real analyses. We address each point below.


W1, Q1. Fairness of baseline comparison

To address concerns about the fairness of our baseline comparisons, we conduct additional analyses to examine the effects of training budgets and PXRD augmentation.

Training budget and optimization. We present the validation match-rate curves of Bragglite and the baselines throughout training.

Figure: Validation match-rate curves during training

As shown in the figure, the baselines generally show diminishing improvements toward the end of their training runs. Importantly, Bragglite already outperforms all baselines at 300K optimizer steps, demonstrating that its advantage cannot be explained by a longer training budget.

To further examine the effect of training budgets and optimization settings, we retrain RealPXRD-Solver, PXRDGen, and XRDiff for 800K steps using the same optimization protocol as Bragglite, while retaining each method's original augmentation strategy. We compare the resulting performance with Bragglite below.

TaskMethodMP-20RRUFFopXRD
StoichiometryRealPXRD-Solver60.51 / 81.6127.33 / 42.6722.62 / 26.79
PXRDGen55.84 / 76.8830.00 / 56.6726.79 / 32.14
XRDiff58.61 / 80.8019.67 / 43.6722.02 / 30.95
Bragglite (Ours)70.85 / 82.5952.40 / 62.6032.14 / 42.50
Element-set + NXRDiff38.36 / 52.6915.00 / 23.3311.31 / 15.48
Bragglite (Ours)47.48 / 71.5138.20 / 52.8023.57 / 28.21

(Entries report Top-1 / Top-20 structure match rates (%); Bragglite results are taken from the main table.)

Despite controlling for training budgets and optimization settings, Bragglite consistently outperforms the retrained baselines across all datasets and task settings, indicating that these factors alone do not account for its performance advantage.

PXRD augmentation. To isolate the effect of augmentation, we additionally train two strong baselines, RealPXRD-Solver and XRDiff, using the same augmentation pipeline as Bragglite.

MethodMP-20 Top-1 (%)RRUFF Top-1 (%)
RealPXRD-Solver (original)56.3527.80 ± 4.76
RealPXRD-Solver (+ Bragglite augmentation)56.5135.50 ± 1.29
XRDiff (original)55.1017.80 ± 0.84
XRDiff (+ Bragglite augmentation)54.4133.25 ± 0.96
Bragglite70.8552.40 ± 0.89

The results show that our augmentation improves the RRUFF match rates of both baselines. Even with the same augmentation, Bragglite maintains a substantial performance advantage over both baselines.


W2, Q2. Prototype-OOD results need more systematic analysis

To assess whether the performance degradation on Prototype OOD is explained by differences in structural complexity, we conducted stratified analyses based on (1) the number of atoms per primitive cell, (2) crystal system, and (3) the number of independent Wyckoff orbits. We report the Top-1 structure match rate separately within each category for both IID and Prototype OOD test sets, as shown in the figure below.

Figure: Top-1 match rate across structural-complexity categories

We observe a consistent and substantial performance gap between IID and Prototype OOD across all categories. Notably, the performance gap remains substantial even for simple structures (89.8% vs. 29.1% for 1–4 atoms; 81.3% vs. 20.2% for 1–2 Wyckoff orbits), suggesting that structural complexity alone cannot explain the poor generalization to unseen prototypes. We additionally provide histograms showing the number of test samples in each bin for both the IID and Prototype OOD splits.

Figure: Test sample distributions across structural-complexity categories

Importantly, the consistently large performance gap across all examined bins indicates that no single structural-complexity factor can explain the degradation, supporting our conclusion that prototype novelty is a key challenge for generalization.

W3, Q3. Some data construction may introduce selection bias

We initially adopted the 128-record cutoff to reduce the preprocessing cost of large-scale structural deduplication and leakage control. This filtering excluded 8,107 unique structures from 138 compositions. However, we acknowledge that it could disproportionately exclude polymorph-rich compositions, potentially affecting the difficulty and representativeness of the benchmark.

To address this concern, we update PXRD-Bench by removing the 128-record cutoff and incorporating the previously excluded structures into the existing splits. We apply the same structural deduplication and leakage-control procedures while preserving the original split criteria. The updated dataset statistics are included in the revised manuscript. The table below summarizes the additional records and resulting dataset sizes for each split.

SplitOriginalAddedPXRD-Bench v2
Train4,107,627+6,7984,114,425
IID · val19,713+27119,984
IID · test19,710+36120,071
Composition OOD · val19,754+4619,800
Composition OOD · test19,735+6019,795
Pair OOD · val19,805+3819,843
Pair OOD · test19,836+3819,874
Prototype OOD · val17,988+26618,254
Prototype OOD · test18,523+22918,752
Total4,262,691+8,1074,270,798

As full retraining is still ongoing, we first report preliminary results obtained by fine-tuning the original checkpoints for an additional 100K steps on the expanded training set.

(Exact stoichiometry)

ModelIIDPair OODPrototype OODComposition OODRRUFFopXRD
PXRDGen61.52 / 88.5762.60 / 88.877.23 / 34.1865.53 / 90.820.00 / 3.001.45 / 7.23
RealPXRD-Solver61.04 / 88.9660.25 / 88.672.73 / 15.6265.23 / 91.7019.40 / 46.8019.04 / 26.99
XRDiff30.37 / 76.0735.16 / 75.880.88 / 10.3532.03 / 80.183.40 / 27.605.54 / 17.35
Crystalyze12.80 / 36.4815.72 / 38.990.27 / 3.4014.46 / 35.500.00 / 2.600.36 / 3.33
Bragglite (Ours)73.54 / 94.3474.12 / 94.3410.06 / 33.5075.29 / 94.9243.60 / 68.8023.41 / 37.83

(Element-set)

ModelIIDPair OODPrototype OODComposition OODRRUFFopXRD
XRDiff20.37 / 45.7319.99 / 45.790.15 / 3.1021.43 / 50.792.11 / 12.523.21 / 9.76
Bragglite (Ours)52.83 / 86.7259.47 / 87.014.88 / 21.3954.10 / 87.7030.40 / 56.8017.11 / 25.06

Full retraining of the baselines is expected to finish before the end of the rebuttal period, and we will update their results accordingly. Bragglite's full retraining is also ongoing and will take longer to complete. We will include its final results in the camera-ready manuscript, along with the updated dataset statistics and evaluation results.


W4, Q4. How much does two-phase recovery performance depend on oracle per-phase atom counts?

To quantify the dependence of two-phase recovery on oracle per-phase atom counts, we evaluate recovery using predicted rather than oracle atom counts. We train a PXRD-conditioned predictor to estimate each phase's atom count from its XDecomposer-generated phase-specific PXRD profile, and use these estimates to condition Bragglite, keeping the rest of the pipeline unchanged.

Count conditioningTop-1 match rateTop-20 match rate
Predicted count0.0%19.0%
Oracle count19.05%47.62%

(For Top-20 matching, we sample five atom-count predictions and generate four candidate structures per prediction, yielding 20 candidates.)

These results highlight atom-count estimation as an important challenge for two-phase recovery, while oracle-count results provide an upper-bound reference for structure recovery. We will include end-to-end evaluation with predicted per-phase atom counts as an additional benchmark track that jointly assesses atom-count estimation and two-phase structure recovery, and release our count predictor as a baseline.

W5, Q5. Can the author provide further evidence that the chosen PXRD representations and augmentation generalize beyond the experimental datasets used here?

To address the reviewer's concern about generalizability beyond RRUFF and opXRD, we additionally evaluate MP-20-trained Bragglite on two external experimental datasets: GALAXI single-phase PXRD and X2SBench. Both datasets were released concurrently with our work and are evaluated without further tuning.

Bragglite settingX2SBench (n=591) Top-1 / Top-20GALAXI (n=12) Top-1 / Top-20MP-20 Top-1 / Top-20
Full augmentation + exact composition36.0% / 53.6%83.3% / 100%70.85% / 82.59%
No augmentation + exact composition0.00% / 2.03%0.00% / 8.33%79.76% / 88.88%
Full augmentation + element-set21.2% / 38.1%66.7% / 83.3%47.48% / 71.51%
No augmentation + element-set0.00% / 0.00%0.00% / 0.00%53.15% / 75.04%

These results support the generalizability of our selected PXRD representation and show that the benefits of full augmentation transfer to both external datasets.

Reviewer_LSmZOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PXRD-Bench, a unified benchmark for crystal structure determination from powder X-ray diffraction (PXRD) patterns. It provides ~4.2M structures paired with simulated PXRD profiles, leakage-controlled OOD splits (holding out composition-prototype pairs, compositions, or structural prototypes), experimental evaluation sets (RRUFF and opXRD), and task extensions beyond known-composition single-phase recovery to element-set conditioning and two-phase mixture recovery. The authors also propose Bragglite, a diffusion Transformer baseline built on Crystalite with a peak-level Transformer encoder and a systematic sim-to-real PXRD augmentation pipeline, achieving 44.2% Top-1 match rate on RRUFF and outperforming existing PXRD-to-structure baselines across settings.

Critical strengths

(Note: My background is in machine learning rather than PXRD crystal structure determination, so my review focuses on the benchmark design, ML methodology, and empirical analysis.)

  1. Comprehensive, Leakage-Controlled Benchmark Design: PXRD-Bench addresses major evaluation gaps in prior work by standardizing stricter structural matching tolerances, removing training structures that match experimental test sets, and separating Pair-OOD, Composition-OOD, and Prototype-OOD splits. Extending the tasks to unknown stoichiometry (element-set + atom count) and two-phase mixtures makes the benchmark much more realistic.

  2. Strong Baseline and Insightful Empirical Findings: Bragglite provides a substantial jump over prior methods (e.g., +24.6% Top-1 on RRUFF over the strongest prior PXRD baseline when trained on PXRD-Bench). More importantly, the paper surfaces valuable diagnostic findings—such as showing that several prior PXRD-conditioned models underperform composition-only CSP baselines on experimental data, and that unseen structural prototypes (Prototype-OOD) remain a major bottleneck across all models.

Critical weaknesses
  1. Sim-to-Real Transfer Still Relies on Dataset-Specific Heuristics and Small Experimental Sets: While the augmentation pipeline improves RRUFF transfer, Figure 5a and Table 3 show that augmentation effectiveness varies across experimental datasets (RRUFF vs. opXRD) and that smaller MP-20 training actually transfers better to RRUFF than the full 4.1M PXRD-Bench unless Si/O mineral structures are explicitly oversampled. Additionally, the experimental test sets remain relatively small (100 for RRUFF, 83 for opXRD, and 21 for GALAXI two-phase mixtures).

  2. Simplifying Oracle Assumptions in Element-Set and Two-Phase Tasks: In both the element-set and two-phase recovery tasks, models are still provided the exact oracle primitive-cell atom counts (N, or N_1 and N_2). Furthermore, the two-phase pipeline relies on a two-stage setup (fine-tuned XDecomposer followed by independent per-phase generation) evaluated mainly on binary metallic systems from MP-20, leaving end-to-end recovery with unknown atom counts open.

Questions for authors
  1. Sensitivity to Unknown Atom Counts (N): In real experimental workflows where exact stoichiometry is unknown, the exact primitive-cell atom count N (or N_1, N_2 in mixtures) is typically also unknown. How sensitive is Bragglite if N is estimated or swept over a range of candidate values?

  2. Scaling Anomaly on Experimental Transfer (MP-20 vs. PXRD-Bench): In Table 1, several baselines (e.g., PXRDGen, Crystalyze) collapse to near 0% Top-1 on RRUFF/opXRD when trained on PXRD-Bench compared to MP-20, and Bragglite also drops from 52.4% (MP-20) to 44.2% (PXRD-Bench) on RRUFF before Si/O oversampling. Why do baselines degrade so severely when scaling up the simulated training corpus?

4 comments

We thank the reviewer for the positive assessment of PXRD-Bench's comprehensive benchmark design, extended task settings, and Bragglite's strong empirical results and insightful analyses. We address each point below.


W1. Sim-to-real transfer still relies on dataset-specific heuristics and small experimental sets.

To address concerns about dataset-specific tuning and small experimental evaluation sets, we additionally evaluate Bragglite on two experimental datasets introduced concurrently with our work, GALAXI single-phase and X2SBench, without further tuning.

Bragglite settingX2SBench (n=591) Top-1 / Top-20GALAXI (n=12) Top-1 / Top-20
Full augmentation + exact composition36.0% / 53.6%83.3% / 100%
No augmentation + exact composition0.00% / 2.03%0.00% / 8.33%
Full augmentation + element-set21.2% / 38.1%66.7% / 83.3%
No augmentation + element-set0.00% / 0.00%0.00% / 0.00%

Augmentation substantially improves performance on both external datasets. Together with our RRUFF and opXRD results, this demonstrates consistent sim-to-real improvements across four experimental datasets. Finally, as a benchmark and analysis paper, PXRD-Bench systematically quantifies how the effectiveness of individual augmentation techniques varies across datasets, identifying this dataset dependence as an important remaining challenge in sim-to-real transfer.

Si/O oversampling. While dataset-specific, Si/O oversampling serves as a controlled analysis of chemical distribution mismatch, rather than a necessary component of our general sim-to-real pipeline. We compare Bragglite trained on PXRD-Bench with and without oversampling on simulated and experimental RRUFF patterns.

Training settingSimulated Top-1Simulated Top-20Experimental Top-1Experimental Top-20
PXRD-Bench48.83 ± 3.1672.92 ± 2.5543.75 ± 2.9570.42 ± 1.74
+ Si/O oversampling56.25 ± 2.0876.25 ± 3.7854.58 ± 3.4274.17 ± 1.14

Improvements in both settings suggest that chemical distribution mismatch, beyond sim-to-real effects alone, contributes to the performance gap between MP-20 and PXRD-Bench on RRUFF.


W2, Q1. Sensitivity to unknown atom counts

To evaluate Bragglite's sensitivity to unknown atom counts, we additionally evaluate structure recovery using predicted rather than oracle atom counts. We train a PXRD-conditioned atom-count predictor and compare structure recovery performance under oracle and predicted counts on MP-20 and RRUFF. For two-phase recovery, we predict each phase's atom count from its XDecomposer-generated PXRD profile and use the predicted counts to condition Bragglite, keeping the rest of the pipeline unchanged.

Element-set recovery. The results are summarized below:

DatasetCount conditioningCount Top-1Count Top-5Structure Match Top-1Structure Match Top-20
MP-20Predicted66.71%89.08%40.08%56.10%
MP-20Oracle——47.48%71.51%
RRUFFPredicted29%70%15.0%37.0%
RRUFFOracle——38.20%52.80%

Two-phase recovery (GALAXI).

Count conditioningAll-phase match Top-1All-phase match Top-20
Predicted count0.0%19.0%
Oracle count19.05%47.62%

(For Top-20 matching, we sample five atom-count predictions and generate four candidate structures per prediction, yielding 20 candidates per PXRD pattern.)

These results highlight atom-count estimation as an important challenge for experimental PXRD, while oracle-count results provide an upper-bound reference for structure recovery. We view end-to-end evaluation with predicted counts as an additional benchmark track that jointly assesses atom-count estimation and structure recovery, and will release our count predictor as a baseline.

Q2. Why do baselines degrade when scaling up the simulated training corpus from MP-20 to PXRD-Bench?

Composition shortcut in MP-20. We hypothesize that the higher experimental performance of MP-20-trained models partly stems from a stronger composition-to-structure shortcut rather than better PXRD conditioning. To investigate this, we compare the number of distinct structures associated with each reduced composition across training datasets.

Figure: Distribution of composition multiplicity

In MP-20, approximately 72% of samples belong to compositions associated with a single structure, compared with only 31% in PXRD-Bench. Thus, composition is substantially more predictive of structure in MP-20, potentially allowing models to achieve high match rates without effectively learning PXRD conditioning.

This hypothesis is supported by composition-only CSP baselines. DiffCSP and Crystalite trained on MP-20 achieve RRUFF Top-1 match rates of 31.60% and 39.20%, respectively. Notably, Crystalite's Top-1 drops from 39.20% to 16.20% when trained on PXRD-Bench, while its Top-20 remains comparable (52.60% vs. 57.80%). This suggests that the composition shortcut in MP-20 particularly benefits Top-1 structure ranking.

Consistent with this explanation, all baselines achieve higher RRUFF Top-1 match rates when trained on MP-20 than on PXRD-Bench. Bragglite exhibits the same trend. Crystalyze follows this trend but has a smaller absolute decline (5.0% to 0.6%), as its performance is already low with MP-20. These results suggest that MP-20's stronger composition-to-structure shortcut contributes to its apparent experimental performance advantage, rather than necessarily reflecting better PXRD conditioning.

PXRDGen. PXRDGen shows a particularly large performance gap between MP-20 and PXRD-Bench training. Our analysis reveals that the MP-20-trained model suffers from PXRD encoder representation collapse and relies primarily on composition priors rather than PXRD inputs. This explains its unexpectedly high RRUFF performance: ignoring PXRD inputs makes the model insensitive to experimental noise.

To verify this, we measure the average change in model outputs when varying PXRD or composition inputs. As shown below, the MP-20-trained model is insensitive to PXRD changes, whereas the PXRD-Bench-trained model responds to both inputs. The near-zero encoder output variance further supports representation collapse.

Training setSame inputChanged PXRDChanged compositionEncoder variance
PXRD-Bench1.59×10⁻⁷0.4820.5300.610
MP-201.54×10⁻⁷1.53×10⁻⁷0.9471.66×10⁻⁷

(Input sensitivity is measured as the average change in model outputs when varying each input while keeping the other fixed. The same-input setting serves as a control.)

We further find that PXRDGen's contrastive pretraining is highly sensitive to data-shuffling order, with four of five seeds resulting in encoder collapse. Retraining with a non-collapsed encoder preserves MP-20 performance but substantially reduces RRUFF performance, further supporting the role of representation collapse in the original model's unexpectedly high RRUFF performance.

PXRDGenMP-20 Top-1RRUFF Top-1 / Top-20
Original (collapsed)54.75%30.80% / 57.00%
Retrained (non-collapsed)58.48%3.00% / 12.00%

Dear Reviewer LSmZ,

As the discussion period is nearing its close, we wanted to briefly follow up on our rebuttal. If our responses have sufficiently addressed your concerns, would you consider raising your rating accordingly? If any concerns remain, we would be happy to clarify them.

Thank you for your time and consideration.

The responses have addressed most of my concerns. Considering the overall contribution, I keep my opinion and reflect it in the rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.