- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose HEABench, a dataset pairing atomic structures with all-electron charge densities calculated using DFT (Density Functional Theory), focusing on high-entropy alloys (HEAs: alloys composed of five or more elements mixed in roughly equal proportions). The dataset consists of 3,109 periodic cells, each containing 5 elements and 100 atoms (60 elements, 1,033 possible combinations of 5 elements), and includes electronic targets derived from the relaxed structure, 3D charge density, formation energy, DOS (density of states), and band-derived electronic targets; of these, 886 configurations include atomic magnetic moments derived from spin-polarized calculations (§3, Table 1, Appendix B.4).
The authors define three tasks based on this data: (i) Global prediction: Predict the physical properties of structural units based on combinations of composition, structure, and charge density ρ, and compare the results with a control where ρ is replaced by a Gaussian field G generated from atomic positions. (ii) Local matching: A bidirectional search that maps structural patches and charge density patches, extracted at the midpoint of the same atomic pair, via supervised learning. By setting conditions that restrict candidates to the same parent configuration and the same reference element (the element at one end of the atomic pair), the authors compare encoders for 4 types of structures × 4 types of charges. (iii) Magnetic transition: We fine-tune the structural encoder pre-trained as described in (ii) using a small number of magnetic labels to predict magnetic moments based solely on structure (§4, Figure 3).
The main results reported by the authors are: (i) that even with G, the majority of the gains from ρ fusion can be achieved (Table 3); (ii) that when candidates are narrowed down to those with the same parent configuration, the differences between encoders widen (Figure 5, Table 4) , (iii) when using labels for 50 configurations, ρ pretraining reduces the site-level MAE (Mean Absolute Error) by 10.9% compared to random initialization (Figure 6, Table 10).
Critical strengths
-
This paper provides a dataset of chemically irregular alloys in which the relaxation structure, 3D charge density, and electronic targets (along with some magnetic labels) are paired using the same DFT procedure. Within the comparison range shown in Table 1, the only existing HEA data with 3D charge density is that of Mirzaee et al. (2024) for a single composition system; even the work by Pathrudkar et al. (npj Comput. Mater. 2025) cited in §2.2 of the main text is limited to a small number of element systems, such as CrFeCoNi. Spanning 60 elements and 1,033 pairs, this dataset is the most extensive collection of irregular alloy data with charge density information. The computational cost of approximately 2.52 million CPU core-hours (Appendix A.1) is on a scale that would be difficult for individual researchers to replicate. Reason for its impact on the overall evaluation: If made publicly available, this dataset would serve as a resource to fill the gap in learning structural–electronic representations for disordered systems. However, this dataset has not yet been made public (Critical Weaknesses 1).
-
The authors designed a control experiment to isolate the contribution of charge density. (i) Even when ρ is replaced with G (a field generated solely from the structure, where a Gaussian function weighted by the number of valence electrons is placed at each atomic position), most of the improvement on the electronic target is reproduced, and the improvement in MAE when G is replaced with ρ is limited to −3.6% to +3.2% (negative values indicate that ρ performs worse) (§5.1, Table 3). The instances where ρ outperforms G are limited to specific cases, such as the NLL (negative log-likelihood, calibrated using validation data) of 0.029 [0.005, 0.062] for the DOS curve. (ii) However, the authors visualized a difference that was not apparent in candidates from a different parent configuration, where R@1 (the proportion of times the correct answer ranks first) saturated at 98–99.9%, using candidates from the same parent configuration (Figure 5). Reason for its impact on the overall evaluation: This comparison serves as a concrete means of correcting the field’s tendency to assume that predictions improve simply by including charge density, and its evaluation design allows for reuse in subsequent research.
Critical weaknesses
-
Reviewers cannot access or verify the dataset, which is the core contribution of this paper, at the time of review (although ICLR does not require public release, the data itself is the main contribution of this paper). First, the anonymous repository contains only a README file, which merely states that the code and segmentation are “being organized for release” and that the data “will be provided separately.” This paper also lacks a Reproducibility Statement, which is strongly recommended by the Author Guidelines. Second, reviewers cannot determine the physical meaning of the labels. Section 4 describes the data used for pre-training as “structure–charge pairs computed without distinguishing electron spins,” but the authors do not indicate whether each configuration in the main dataset was computed using non-spin-polarized (NSP) calculations. Many configurations contain magnetic elements (Fe, Co, Ni, Mn, Cr) (Figure 2a), and if they were NSP, those labels would not correspond to the magnetic ground state. This paper also lacks any description of the k-point mesh or lattice type. Reason for impact on the overall evaluation: Neither the reviewers nor the AC can verify the value and reliability of the data at the time of review.
-
The authors’ claims regarding the value of charge density cannot be confirmed based on current experiments. The Abstract (p. 1) positions charge density as “a source of transferable supervision.” First, the authors do not specify the number of training seeds, and the 95% confidence intervals are based solely on resampling of the test dataset and do not include training random numbers. Consequently, reviewers cannot determine whether the increase attributed to ρ in (i) or the improvement in (iii) (Tables 10 and 11) exceeds the variation between seeds. Second, the only control for (ii), the Gaussian radial reference without learning, involves matching a Gaussian field placed at the atomic positions with ρ in the radial direction only (Appendix B.1); under the condition of selecting from four candidates, this already reaches R@1 of 77.93% (25% by chance) (Table 4). The trained encoder outperforms this in 13 out of 16 pairs (by a maximum of +21.8 points); however, since the reference discards angular information, this improvement could be explained by atomic configurations that include angular information, and the authors have not demonstrated that the trained encoder utilizes electronic information beyond geometry. Reason for impact on the overall evaluation: The basis for the central scientific claim of this paper remains unconfirmed.
Questions for authors
-
[Clarification] The anonymous repository currently contains only a README; can the authors provide the data (at least the samples and splits) and code during the peer review period? Please also specify the license, format (whether the data is in the form of the charge density files CHGCAR output by the DFT program VASP used in the paper, or a resampled version), total size, hosting location, and maintenance plan. (Critical Weaknesses 1)
-
[Clarification] Were the individual configurations in the main dataset calculated using non-spin-polarized (NSP) or spin-polarized (SP) methods? Although Appendix B.4 defines the NSP/SP notation, there is no corresponding column in Table 7; thus, the reviewers cannot understand the relationship between the 2,446 configurations in (ii) and the 1,993 NSP configurations used for pre-training in (iii), as well as the 3,109 configurations. For the NSP case, can you demonstrate the differences in structure, charge density, and formation energy compared to spin-polarized calculations for some configurations? The reviewer also requests a description of the distinction between the use of k-point meshes, lattice types and cell shapes, and the use of random substitution versus SQS (a configuration that reproduces the statistics of random alloys in finite cells). (Critical Weaknesses 1)
-
[Clarification] How many training seeds were used for each experiment? When (i) (Table 3) and (iii) (Tables 10 and 11) are rerun with 3–5 seeds, do the specific increase in ρ and the difference in ρ with respect to G under random initialization with 50 labels exceed the variation between seeds? (Critical Weaknesses 2)
-
[Clarification] Since the Gaussian radial reference without training in Table 4 already reaches R@1 of 77.93%, it is possible that much of the matching in (ii) is determined by atomic configurations. If matching is learned using patches of differential charge density, obtained by subtracting the superposition of charge densities of isolated atoms from ρ (this superposition can be constructed without iterative DFT calculations, as it corresponds to the electron rearrangement resulting from interatomic bonds), (using the best combination, for example, just the pair of EquiformerV2, which has the highest top-up in Table 4, and the authors’ implementation of the charge density encoder LSHCE), how much do R@1 and MRR (the average of the reciprocals of the ranks of correct answers) exceed the levels achieved by chance (25% for R@1 and approximately 52% for MRR) or the radial reference recalculated using the same difference charge density? If it outperforms them, it can be said that (ii) captures electronic rearrangements beyond the geometric level. (Critical Weaknesses 2)
-
[Clarification] Do the 1,993 NSP configurations used for pre-training in (iii) and the 133 configurations used for the magnetic prediction test share the same structure or the same set of five elements? (Critical Weaknesses 2)
-
[Proposal] (Proposal for improvement that does not affect the evaluation) Since the comparison in (iii) currently relies only on random initialization and G pre-training, the reviewer proposes fine-tuning using CHGNet (Deng et al., Nat. Mach. Intell. 2023; pre-trained to include magnetic moments) or fine-tuning from publicly available pre-trained models such as EquiformerV2 and MACE, and extending the G pre-training to 300 and 620 labels, the practical significance of structure–charge density pre-training will become clearer.
-
[Proposal] (Proposed improvement that does not affect evaluation) The current partition is based on parent structural units, meaning the same set of five elements may span both training and testing. Providing a standard partition based on sets of five elements (leave-element-set-out) or unknown elements would create a benchmark for measuring composition extrapolation, the most critical challenge in HEA.
-
[Clarification] Did the authors, in (i), downscale ρ (median of approximately 3.3 million grid points) to the same 56×40×40 grid as G before inputting it? If so, fine-grained information in ρ may have been lost, potentially resulting in a smaller difference compared to G. (Critical Weaknesses 2)
11 comments
Spin settings and physical labels (Weakness 1; Question 2)
The full dataset contains both non-spin-polarized (NSP) and spin-polarized (SP) calculations. In T3, pretraining uses NSP records and magnetic prediction uses SP records. The different subset sizes reflect the data available at each experimental stage, as summarized below.
\text{Dataset or stage}&\text{Parent configurations}&\text{Spin setting}\\\\ \hline \text{Full dataset / global prediction}&3,109&\text{NSP and SP}\\\\ \text{Local structure-charge matching}&2,446&\text{NSP and SP}\\\\ \text{Transfer pretraining pool}&1,993&\text{NSP (1,595/199/199 training/validation/test).}\\\\ \text{Magnetic prediction}&886&\text{SP (620/133/133 training/validation/test). Nested training budgets: 50, 300, 620.}\\\\ \end{array}$$ NSP labels describe the spin-unpolarized reference state. The magnetic targets come from converged SP calculations, with `MAGMOM` used to initialize the moments. The learned charge field is the total density ($\rho_{\uparrow} + \rho_{\downarrow}$), while magnetic information is associated with the spin difference. Only the structure encoder is transferred, so magnetic prediction requires structures at inference. The starting structures are 100-atom, FCC-derived cells ($5 \times 5 \times 4$, reference lattice parameter $3.59\\,\text{Å}$), with occupations generated by random substitution or SQS optimization using ICET. Cells and atomic positions are then relaxed. Calculations use VASP 6.3.2 with PAW-PBE and a $550\\,\mathrm{eV}$ cutoff. Relaxation and static calculations use a $1 \times 1 \times 1$ $k$-point mesh, while band calculations use the line-mode paths recorded in the `KPOINTS` files. We have prepared an NSP/SP structure and charge-density comparison for the same alloy composition ($\mathrm{Mn}_{19}\mathrm{Fe}_{21}\mathrm{Co}_{22}\mathrm{Cu}_{21}\mathrm{Zn}_{17}$). The main atom-centered density features are similar, with local differences between the relaxed states. The structures were relaxed independently and then aligned, so those differences include both relaxation and the response to spin polarization **This distinction gives the transfer experiment a concrete physical interpretation: its source supervision is the NSP electronic environment, while its downstream target is SP magnetism**. Only the structure encoder is transferred, and downstream prediction uses atomic structures at inference. The results in Section 3 directly test whether that source representation helps learning the target property.Repeated training and charge-density gains – T1 (Weakness 2; Question 3)
We have repeated the core T1 comparisons and each T3 initialization at every label budget three times. T3 keeps the pretrained weights and data splits fixed. T2 uses one checkpoint per encoder pair, and the original composition-only reference and original diagnostic plots retain their single-run predictions. All values below are sample standard deviations (SD) across the three runs.
T1: Charge-density prediction
Real charge density lowers mean MAE and RMSE relative to the matched Gaussian field on all seven targets. The strongest repeated-run evidence is on formation energy and the two count targets, where both metrics improve in every run. The remaining targets improve in MAE in two of the three runs. Thus, the mean benefit is broad, and its consistency across runs is clearest for these three targets.
The comparison uses the same fusion model and training settings, replacing only the Gaussian field with charge density. Each cell gives MAE / RMSE in the target's original units (eV/atom for formation energy, eV for band energies).
\text{Target}&\text{Gaussian: MAE / RMSE}&\text{Charge: MAE / RMSE}&\text{Mean reduction: MAE / RMSE (\\%)}\\\\ \hline \text{Formation energy}&0.0523\pm0.0025/0.1527\pm0.0105&0.0486\pm0.0041/0.1307\pm0.0103&7.17/14.37\\\\ \text{DOS at the Fermi level}&0.0955\pm0.0175/0.1444\pm0.0332&0.0882\pm0.0042/0.1259\pm0.0115&7.69/12.79\\\\ \text{Crossing count}&0.8087\pm0.0785/1.1180\pm0.1128&0.7595\pm0.0484/1.0350\pm0.0914&6.09/7.42\\\\ \text{Event count}&2.5515\pm0.2718/3.3304\pm0.3827&2.3359\pm0.1898/3.0647\pm0.3309&8.45/7.98\\\\ \text{Crossing slope}&0.0753\pm0.0005/0.1163\pm0.0014&0.0739\pm0.0011/0.1149\pm0.0044&1.87/1.24\\\\ \text{DOS curve}&0.0862\pm0.0067/0.1443\pm0.0161&0.0823\pm0.0005/0.1364\pm0.0003&4.43/5.43\\\\ \text{Band energies}&0.0153\pm0.0002/0.0226\pm0.0007&0.0151\pm0.0001/0.0222\pm0.0004&1.03/1.76\\\\ \end{array}$$ Repeating the original experiment with resampled inputs also supports the original spatial-encoding conclusion: Gaussian and charge inputs both improve on composition-plus-structure inputs for all six electronic targets in every paired run. The added environment analysis finds a DOS-curve benefit in all three local-composition groups (low, middle, and high deviation from the parent composition). Together, these results show that spatial encoding explains much of the fusion benefit, while real charge density provides additional predictive information.
Data access and format (Weakness 1; Question 1)
Thank you for recognizing the value of the paired data and Gaussian-field controls. We have added repeated training, input-resolution comparisons, stronger retrieval controls, and magnetic-transfer experiments across label budgets. Below we address what these experiments show about the data and the value of charge-density supervision.
We apologize that the materials were not sufficiently accessible during the initial review. The code, fixed splits, and running instructions are available at https://anonymous.4open.science/r/HEABENCH-F6E1. The repository also contains three complete sample packages and a lightweight HTML gallery (static images of relaxed structures.). The samples include original VASP outputs, including
CHGCAR, structures, DOS, and eigenvalues. We would add a short reproducibility statement in a future revision.