acceptodds
Under review as a conference paper at ICLR 2027

HEABench:A Multimodal Dataset and Benchmark for High-Entropy Alloy Understanding

Abstract

High-entropy alloys (HEAs) offer a rich compositional space for materials design, yet their heterogeneous atomic environments make learning structure–property relationships challenging. Progress is constrained in part by limited paired atomic and electronic data and by the lack of consistent protocols for evaluating representations across spatial scales. We introduce HEABench, a multimodal dataset and benchmark of 3,109 quinary, 100-atom configurations spanning 60 elements, with paired structures, total electron charge densities, and property targets, complemented by magnetic labels. HEABench supports global property prediction, local structure–charge matching, and site-resolved magnetic transfer under limited supervision. Experiments show that representing atomic positions as spatial fields via site-centered Gaussians recovers much of the electronic-target fusion benefit, whereas additional charge-density gains depend on the target and evaluation metric. Local matching under configuration and elemental identity controls reveals encoder differences obscured by near-saturated cross-configuration retrieval. With 50 magnetically labeled configurations, structure–charge pretraining reduces site-moment MAE by 10.9% relative to random initialization, using only atomic structures during inference. These findings highlight charge density as a source of transferable supervision beyond its target-dependent benefits as an additional prediction input, and establish HEABench as a testbed for learning structural and electronic representations in chemically disordered alloys. Project page: https://anonymous.4open.science/r/HEABENCH-F6E1.

Primary area:
datasets and benchmarks
Keywords:
HEAs, Dataset, Multimodal Representation, Electronic structure

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Qv1aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose HEABench, a dataset pairing atomic structures with all-electron charge densities calculated using DFT (Density Functional Theory), focusing on high-entropy alloys (HEAs: alloys composed of five or more elements mixed in roughly equal proportions). The dataset consists of 3,109 periodic cells, each containing 5 elements and 100 atoms (60 elements, 1,033 possible combinations of 5 elements), and includes electronic targets derived from the relaxed structure, 3D charge density, formation energy, DOS (density of states), and band-derived electronic targets; of these, 886 configurations include atomic magnetic moments derived from spin-polarized calculations (§3, Table 1, Appendix B.4).

The authors define three tasks based on this data: (i) Global prediction: Predict the physical properties of structural units based on combinations of composition, structure, and charge density ρ, and compare the results with a control where ρ is replaced by a Gaussian field G generated from atomic positions. (ii) Local matching: A bidirectional search that maps structural patches and charge density patches, extracted at the midpoint of the same atomic pair, via supervised learning. By setting conditions that restrict candidates to the same parent configuration and the same reference element (the element at one end of the atomic pair), the authors compare encoders for 4 types of structures × 4 types of charges. (iii) Magnetic transition: We fine-tune the structural encoder pre-trained as described in (ii) using a small number of magnetic labels to predict magnetic moments based solely on structure (§4, Figure 3).

The main results reported by the authors are: (i) that even with G, the majority of the gains from ρ fusion can be achieved (Table 3); (ii) that when candidates are narrowed down to those with the same parent configuration, the differences between encoders widen (Figure 5, Table 4) , (iii) when using labels for 50 configurations, ρ pretraining reduces the site-level MAE (Mean Absolute Error) by 10.9% compared to random initialization (Figure 6, Table 10).

Critical strengths
  1. This paper provides a dataset of chemically irregular alloys in which the relaxation structure, 3D charge density, and electronic targets (along with some magnetic labels) are paired using the same DFT procedure. Within the comparison range shown in Table 1, the only existing HEA data with 3D charge density is that of Mirzaee et al. (2024) for a single composition system; even the work by Pathrudkar et al. (npj Comput. Mater. 2025) cited in §2.2 of the main text is limited to a small number of element systems, such as CrFeCoNi. Spanning 60 elements and 1,033 pairs, this dataset is the most extensive collection of irregular alloy data with charge density information. The computational cost of approximately 2.52 million CPU core-hours (Appendix A.1) is on a scale that would be difficult for individual researchers to replicate. Reason for its impact on the overall evaluation: If made publicly available, this dataset would serve as a resource to fill the gap in learning structural–electronic representations for disordered systems. However, this dataset has not yet been made public (Critical Weaknesses 1).

  2. The authors designed a control experiment to isolate the contribution of charge density. (i) Even when ρ is replaced with G (a field generated solely from the structure, where a Gaussian function weighted by the number of valence electrons is placed at each atomic position), most of the improvement on the electronic target is reproduced, and the improvement in MAE when G is replaced with ρ is limited to −3.6% to +3.2% (negative values indicate that ρ performs worse) (§5.1, Table 3). The instances where ρ outperforms G are limited to specific cases, such as the NLL (negative log-likelihood, calibrated using validation data) of 0.029 [0.005, 0.062] for the DOS curve. (ii) However, the authors visualized a difference that was not apparent in candidates from a different parent configuration, where R@1 (the proportion of times the correct answer ranks first) saturated at 98–99.9%, using candidates from the same parent configuration (Figure 5). Reason for its impact on the overall evaluation: This comparison serves as a concrete means of correcting the field’s tendency to assume that predictions improve simply by including charge density, and its evaluation design allows for reuse in subsequent research.

Critical weaknesses
  1. Reviewers cannot access or verify the dataset, which is the core contribution of this paper, at the time of review (although ICLR does not require public release, the data itself is the main contribution of this paper). First, the anonymous repository contains only a README file, which merely states that the code and segmentation are “being organized for release” and that the data “will be provided separately.” This paper also lacks a Reproducibility Statement, which is strongly recommended by the Author Guidelines. Second, reviewers cannot determine the physical meaning of the labels. Section 4 describes the data used for pre-training as “structure–charge pairs computed without distinguishing electron spins,” but the authors do not indicate whether each configuration in the main dataset was computed using non-spin-polarized (NSP) calculations. Many configurations contain magnetic elements (Fe, Co, Ni, Mn, Cr) (Figure 2a), and if they were NSP, those labels would not correspond to the magnetic ground state. This paper also lacks any description of the k-point mesh or lattice type. Reason for impact on the overall evaluation: Neither the reviewers nor the AC can verify the value and reliability of the data at the time of review.

  2. The authors’ claims regarding the value of charge density cannot be confirmed based on current experiments. The Abstract (p. 1) positions charge density as “a source of transferable supervision.” First, the authors do not specify the number of training seeds, and the 95% confidence intervals are based solely on resampling of the test dataset and do not include training random numbers. Consequently, reviewers cannot determine whether the increase attributed to ρ in (i) or the improvement in (iii) (Tables 10 and 11) exceeds the variation between seeds. Second, the only control for (ii), the Gaussian radial reference without learning, involves matching a Gaussian field placed at the atomic positions with ρ in the radial direction only (Appendix B.1); under the condition of selecting from four candidates, this already reaches R@1 of 77.93% (25% by chance) (Table 4). The trained encoder outperforms this in 13 out of 16 pairs (by a maximum of +21.8 points); however, since the reference discards angular information, this improvement could be explained by atomic configurations that include angular information, and the authors have not demonstrated that the trained encoder utilizes electronic information beyond geometry. Reason for impact on the overall evaluation: The basis for the central scientific claim of this paper remains unconfirmed.

Questions for authors
  1. [Clarification] The anonymous repository currently contains only a README; can the authors provide the data (at least the samples and splits) and code during the peer review period? Please also specify the license, format (whether the data is in the form of the charge density files CHGCAR output by the DFT program VASP used in the paper, or a resampled version), total size, hosting location, and maintenance plan. (Critical Weaknesses 1)

  2. [Clarification] Were the individual configurations in the main dataset calculated using non-spin-polarized (NSP) or spin-polarized (SP) methods? Although Appendix B.4 defines the NSP/SP notation, there is no corresponding column in Table 7; thus, the reviewers cannot understand the relationship between the 2,446 configurations in (ii) and the 1,993 NSP configurations used for pre-training in (iii), as well as the 3,109 configurations. For the NSP case, can you demonstrate the differences in structure, charge density, and formation energy compared to spin-polarized calculations for some configurations? The reviewer also requests a description of the distinction between the use of k-point meshes, lattice types and cell shapes, and the use of random substitution versus SQS (a configuration that reproduces the statistics of random alloys in finite cells). (Critical Weaknesses 1)

  3. [Clarification] How many training seeds were used for each experiment? When (i) (Table 3) and (iii) (Tables 10 and 11) are rerun with 3–5 seeds, do the specific increase in ρ and the difference in ρ with respect to G under random initialization with 50 labels exceed the variation between seeds? (Critical Weaknesses 2)

  4. [Clarification] Since the Gaussian radial reference without training in Table 4 already reaches R@1 of 77.93%, it is possible that much of the matching in (ii) is determined by atomic configurations. If matching is learned using patches of differential charge density, obtained by subtracting the superposition of charge densities of isolated atoms from ρ (this superposition can be constructed without iterative DFT calculations, as it corresponds to the electron rearrangement resulting from interatomic bonds), (using the best combination, for example, just the pair of EquiformerV2, which has the highest top-up in Table 4, and the authors’ implementation of the charge density encoder LSHCE), how much do R@1 and MRR (the average of the reciprocals of the ranks of correct answers) exceed the levels achieved by chance (25% for R@1 and approximately 52% for MRR) or the radial reference recalculated using the same difference charge density? If it outperforms them, it can be said that (ii) captures electronic rearrangements beyond the geometric level. (Critical Weaknesses 2)

  5. [Clarification] Do the 1,993 NSP configurations used for pre-training in (iii) and the 133 configurations used for the magnetic prediction test share the same structure or the same set of five elements? (Critical Weaknesses 2)

  6. [Proposal] (Proposal for improvement that does not affect the evaluation) Since the comparison in (iii) currently relies only on random initialization and G pre-training, the reviewer proposes fine-tuning using CHGNet (Deng et al., Nat. Mach. Intell. 2023; pre-trained to include magnetic moments) or fine-tuning from publicly available pre-trained models such as EquiformerV2 and MACE, and extending the G pre-training to 300 and 620 labels, the practical significance of structure–charge density pre-training will become clearer.

  7. [Proposal] (Proposed improvement that does not affect evaluation) The current partition is based on parent structural units, meaning the same set of five elements may span both training and testing. Providing a standard partition based on sets of five elements (leave-element-set-out) or unknown elements would create a benchmark for measuring composition extrapolation, the most critical challenge in HEA.

  8. [Clarification] Did the authors, in (i), downscale ρ (median of approximately 3.3 million grid points) to the same 56×40×40 grid as G before inputting it? If so, fine-grained information in ρ may have been lost, potentially resulting in a smaller difference compared to G. (Critical Weaknesses 2)

11 comments
AuthorsOpenReview ↗

Data access and format (Weakness 1; Question 1)

Thank you for recognizing the value of the paired data and Gaussian-field controls. We have added repeated training, input-resolution comparisons, stronger retrieval controls, and magnetic-transfer experiments across label budgets. Below we address what these experiments show about the data and the value of charge-density supervision.

We apologize that the materials were not sufficiently accessible during the initial review. The code, fixed splits, and running instructions are available at https://anonymous.4open.science/r/HEABENCH-F6E1. The repository also contains three complete sample packages and a lightweight HTML gallery (static images of relaxed structures.). The samples include original VASP outputs, including CHGCAR, structures, DOS, and eigenvalues. We would add a short reproducibility statement in a future revision.

AuthorsOpenReview ↗

Spin settings and physical labels (Weakness 1; Question 2)

The full dataset contains both non-spin-polarized (NSP) and spin-polarized (SP) calculations. In T3, pretraining uses NSP records and magnetic prediction uses SP records. The different subset sizes reflect the data available at each experimental stage, as summarized below.

\text{Dataset or stage}&\text{Parent configurations}&\text{Spin setting}\\\\ \hline \text{Full dataset / global prediction}&3,109&\text{NSP and SP}\\\\ \text{Local structure-charge matching}&2,446&\text{NSP and SP}\\\\ \text{Transfer pretraining pool}&1,993&\text{NSP (1,595/199/199 training/validation/test).}\\\\ \text{Magnetic prediction}&886&\text{SP (620/133/133 training/validation/test). Nested training budgets: 50, 300, 620.}\\\\ \end{array}$$ NSP labels describe the spin-unpolarized reference state. The magnetic targets come from converged SP calculations, with `MAGMOM` used to initialize the moments. The learned charge field is the total density ($\rho_{\uparrow} + \rho_{\downarrow}$), while magnetic information is associated with the spin difference. Only the structure encoder is transferred, so magnetic prediction requires structures at inference. The starting structures are 100-atom, FCC-derived cells ($5 \times 5 \times 4$, reference lattice parameter $3.59\\,\text{Å}$), with occupations generated by random substitution or SQS optimization using ICET. Cells and atomic positions are then relaxed. Calculations use VASP 6.3.2 with PAW-PBE and a $550\\,\mathrm{eV}$ cutoff. Relaxation and static calculations use a $1 \times 1 \times 1$ $k$-point mesh, while band calculations use the line-mode paths recorded in the `KPOINTS` files. We have prepared an NSP/SP structure and charge-density comparison for the same alloy composition ($\mathrm{Mn}_{19}\mathrm{Fe}_{21}\mathrm{Co}_{22}\mathrm{Cu}_{21}\mathrm{Zn}_{17}$). The main atom-centered density features are similar, with local differences between the relaxed states. The structures were relaxed independently and then aligned, so those differences include both relaxation and the response to spin polarization **This distinction gives the transfer experiment a concrete physical interpretation: its source supervision is the NSP electronic environment, while its downstream target is SP magnetism**. Only the structure encoder is transferred, and downstream prediction uses atomic structures at inference. The results in Section 3 directly test whether that source representation helps learning the target property.
AuthorsOpenReview ↗

Repeated training and charge-density gains – T1 (Weakness 2; Question 3)

We have repeated the core T1 comparisons and each T3 initialization at every label budget three times. T3 keeps the pretrained weights and data splits fixed. T2 uses one checkpoint per encoder pair, and the original composition-only reference and original diagnostic plots retain their single-run predictions. All ±\pm values below are sample standard deviations (SD) across the three runs.

T1: Charge-density prediction

Real charge density lowers mean MAE and RMSE relative to the matched Gaussian field on all seven targets. The strongest repeated-run evidence is on formation energy and the two count targets, where both metrics improve in every run. The remaining targets improve in MAE in two of the three runs. Thus, the mean benefit is broad, and its consistency across runs is clearest for these three targets.

The comparison uses the same fusion model and training settings, replacing only the Gaussian field with charge density. Each cell gives MAE / RMSE in the target's original units (eV/atom for formation energy, eV for band energies).

\text{Target}&\text{Gaussian: MAE / RMSE}&\text{Charge: MAE / RMSE}&\text{Mean reduction: MAE / RMSE (\\%)}\\\\ \hline \text{Formation energy}&0.0523\pm0.0025/0.1527\pm0.0105&0.0486\pm0.0041/0.1307\pm0.0103&7.17/14.37\\\\ \text{DOS at the Fermi level}&0.0955\pm0.0175/0.1444\pm0.0332&0.0882\pm0.0042/0.1259\pm0.0115&7.69/12.79\\\\ \text{Crossing count}&0.8087\pm0.0785/1.1180\pm0.1128&0.7595\pm0.0484/1.0350\pm0.0914&6.09/7.42\\\\ \text{Event count}&2.5515\pm0.2718/3.3304\pm0.3827&2.3359\pm0.1898/3.0647\pm0.3309&8.45/7.98\\\\ \text{Crossing slope}&0.0753\pm0.0005/0.1163\pm0.0014&0.0739\pm0.0011/0.1149\pm0.0044&1.87/1.24\\\\ \text{DOS curve}&0.0862\pm0.0067/0.1443\pm0.0161&0.0823\pm0.0005/0.1364\pm0.0003&4.43/5.43\\\\ \text{Band energies}&0.0153\pm0.0002/0.0226\pm0.0007&0.0151\pm0.0001/0.0222\pm0.0004&1.03/1.76\\\\ \end{array}$$ Repeating the original experiment with resampled inputs also supports the original spatial-encoding conclusion: Gaussian and charge inputs both improve on composition-plus-structure inputs for all six electronic targets in every paired run. The added environment analysis finds a DOS-curve benefit in all three local-composition groups (low, middle, and high deviation from the parent composition). Together, these results show that spatial encoding explains much of the fusion benefit, while real charge density provides additional predictive information.
Reviewer_5wxLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

With datasets and models for molecules and crystal structures now well established, the research community is turning toward more complex systems such as high-entropy alloys (HEAs). In this context, this paper presents HEABench, a timely multimodal dataset and benchmark for HEAs. The dataset consists of approximately 3,100 five-element, 100-atom configurations spanning 60 elements. Its key distinguishing feature is that it jointly provides relaxed structures, charge densities, and a range of material properties, whereas existing datasets each provide only some of them. On this benchmark, the authors conduct experiments on three tasks, namely global property prediction, local structure-charge density matching, and low-label magnetic moment transfer, and report several observations.

Critical strengths

The proposed benchmark provides relaxed structures together with charge densities for HEAs, both of which are expensive to compute, making it a valuable resource for the community.

Critical weaknesses

It is unclear why the research questions and the accompanying experiments need to be addressed on the proposed benchmark, since they are not specific to HEAs and could also be studied using existing datasets of other material systems. For a new benchmark, standard evaluations with established models would provide more useful reference results.

Questions for authors

I have the following questions and suggestions, which I believe could strengthen the work:

  • The anonymous link provided in the abstract does not open. Could the authors check it?
  • Before addressing the three tasks considered in the paper, I think it would be informative to first examine how well existing models perform on the proposed benchmark. For example, formation energy prediction would be a natural starting point, as formation energy is a representative target of the models used in the paper (e.g., MACE and EquiformerV2). Additionally, it would be helpful to examine how well models pretrained on general materials datasets transfer to HEA data, and how much fine-tuning improves their performance. Such evaluations would clarify whether the benchmark is genuinely challenging and would serve as reference results for future work.
  • Related to the above comment, the backbones used in the experiments vary across tasks. Except for the local structure-charge density matching task, each task is evaluated with only a limited set of backbones, which makes it difficult to assess the effect of backbone choice comprehensively. Evaluating a consistent set of backbones across all tasks would provide more informative reference results.
  • Beyond its scale and the range of data it provides, does the proposed dataset offer advantages over existing ones from a materials science or physicochemical perspective? For instance, does it include practically relevant alloys or physically interesting configurations that existing datasets do not contain?
  • How many random seeds were used in the experiments?
  • What do the authors consider to be the ultimate challenge that this benchmark and dataset are intended to address?

In addition, I have the following minor suggestions:

  • Table 1 could also compare the computational methodology of each dataset, such as the level of theory.
  • The font size in Table 3 could be slightly reduced to fit the layout.
  • Including more examples similar to Figure 7 would be helpful to readers.

Overall, this work provides a high-quality benchmark for HEAs, which I view positively. However, I have concerns about the experiments accompanying the benchmark. I therefore recommend a weak rejection at this stage, but I am willing to reconsider my score based on the rebuttal. Given the short rebuttal period, I do not expect all of the suggested experiments to be completed, but I would appreciate responses that substantively address the main concerns.


Post-rebuttal update: The additional experiments and revisions in the rebuttal have addressed most of my concerns, and I have updated my score accordingly.

10 comments
AuthorsOpenReview ↗

Access to review materials (Question 1)

We thank the reviewer for the questions on standard prediction baselines, training variability, and the scientific purpose of the dataset. We have added the formation-energy comparisons and the repeated-training experiments described below. They provide more concrete reference results for HEABench and strengthen our original evidence that useful structural and electronic representations can be learned for disordered alloys.

We apologize for the interruption to the anonymous link. The code, fixed splits, and running instructions are available at https://anonymous.4open.science/r/HEABENCH-F6E1/ We have also included three complete sample packages and a lightweight HTML gallery (static images of the relaxed structures.) in the repository. The samples contain original calculation outputs, including CHGCAR files.

AuthorsOpenReview ↗

Formation-energy baselines and transfer (Question 2)

We have added MACE and CHGNet comparisons covering zero-shot prediction, fine-tuning, and training from scratch. Fine-tuning substantially improves both models over zero-shot prediction. It also outperforms training the same backbone from scratch (MAE reductions of about 42% for MACE and 18% for CHGNet), with the improvement present in all three paired runs. This shows that general materials pretraining provides useful information for the HEA task, and also supports HEABench's value as a benchmark for learning and transfer in chemically disordered alloys.

\text{Model}&\text{Initialization}&\text{MAE}&\text{RMSE}\\\\ \hline \text{MACE-MP-0b3 medium}&\text{Zero-shot}&148.50&394.57\\\\ \text{MACE-MP-0b3 medium}&\text{Fine-tuned}&58.30\pm3.13&127.53\pm12.39\\\\ \text{MACE-MP-0b3 medium}&\text{Scratch}&100.46\pm4.75&207.99\pm1.72\\\\ \text{CHGNet 0.3.0}&\text{Zero-shot}&170.03&404.37\\\\ \text{CHGNet 0.3.0}&\text{Fine-tuned}&49.35\pm1.24&133.47\pm1.86\\\\ \text{CHGNet 0.3.0}&\text{Scratch}&60.40\pm1.36&157.85\pm10.16\\\\ \end{array}$$ All results use the same 300 test configurations. Errors are in meV/atom, and fine-tuning and Scratch report the mean $\pm$ sample standard deviation (SD) over three runs. Within each backbone, fine-tuning and Scratch use the same data, energy head, elemental baseline, and training budget (140 epochs, with model selection based on validation performance). Zero-shot evaluation uses the pretrained model's own elemental references without fitting to HEA labels.
AuthorsOpenReview ↗

Training repeats and stability (Question 3)

The added core prediction and transfer comparisons use three training runs, and the counts are summarized below.

\text{Experiment}&\text{Training repetitions and evaluation}\\\\ \hline \text{Core T1 input comparisons}&\text{Three per condition. The original composition-only reference and original diagnostic plots use one run.}\\\\ \text{MACE/CHGNet formation energy}&\text{Three fine-tuning runs and three Scratch runs per backbone.}\\\\ \text{T2 local retrieval}&\text{One trained checkpoint per encoder pair.}\\\\ \text{T3 magnetic transfer}&\text{Three downstream runs per initialization and budget (50, 300, 620). Pretrained weights and data splits are fixed.}\\\\ \end{array}$$ **For T1, the repeated runs support both the original spatial-encoding result and the additional value of real charge density.** On the original resampled inputs, Gaussian and charge fields both improve on composition-plus-structure inputs for all six electronic targets in every paired run. With complete charge-grid inputs, real charge density lowers mean MAE and RMSE relative to the matched Gaussian field on all seven targets. The gains are most consistent for formation energy and the two count targets (both metrics improve in all three runs, and the other targets improve in MAE in two of three runs). **For T3, charge pretraining gives stable site-MAE improvements at 300 and 620 labels, and directly supports the original claim of transferable electronic supervision and improved magnetic-label efficiency.** At 300 labels, it reaches almost the same accuracy as Scratch with 620 labels, using roughly half as many labeled configurations. At 620 labels, its mean site MAE and RMSE are lower than Scratch by 6.0% and 4.4%. Table gives site MAE / RMSE as mean $\pm$ SD. All initializations use the same structure encoder and labeled data within each budget. $$\begin{array}{rlr} \text{Label budget}&\text{Initialization}&\text{Site MAE / RMSE (}\mu_B\text{)}\\\\ \hline 50&\text{Scratch}&0.2363\pm0.0096/0.5478\pm0.0109\\\\ 50&\text{Gaussian pretraining}&0.2287\pm0.0096/0.4991\pm0.0227\\\\ 50&\text{Charge pretraining}&0.2435\pm0.0658/0.5496\pm0.1276\\\\ 300&\text{Scratch}&0.1675\pm0.0020/0.4276\pm0.0049\\\\ 300&\text{Gaussian pretraining}&0.1531\pm0.0021/0.3988\pm0.0035\\\\ 300&\text{Charge pretraining}&0.1507\pm0.0002/0.3989\pm0.0008\\\\ 620&\text{Scratch}&0.1493\pm0.0011/0.4012\pm0.0028\\\\ 620&\text{Gaussian pretraining}&0.1466\pm0.0116/0.3999\pm0.0319\\\\ 620&\text{Charge pretraining}&0.1403\pm0.0008/0.3835\pm0.0011\\\\ \end{array}$$ With only 50 labels, the benefit is clearest for low-moment sites: their errors are roughly halved relative to Gaussian pretraining (MAE/RMSE reductions of 46.4%/49.5%). Larger moments are more sensitive to training variation, so all-site improvement is not consistent across runs. This is consistent with the NSP representation still having a strong influence before the model has fully adapted to SP magnetism. **The original 10.9% improvement remains a valid result for the original run, while the additional runs and budgets show a consistent low-moment benefit and stable overall site-level gains with more labels.**
Reviewer_euysOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HEABench, a multimodal dataset and benchmark comprising 3,109 quinary, 100-atom configurations spanning 60 elements, with paired atomic structures, total electron charge densities, and property targets. It evaluates structural and electronic representations across three tasks: global multimodal probing, local structure–charge matching, and low-label magnetic transfer, using models such as eqv2, CHGNet, and MACE for local matching.

Critical strengths
  1. HEAbench provides dataset for local electronic representation learning and cross-modal pretraining, reducing the computational cost for the community to conduct related research. Table 1 also provides a clear comparison between HEAbench and existing datasets.
  2. The low-label magnetic transfer experiments effectively demonstrate the downstream utility of HEABench.
Critical weaknesses
  1. I still have concerns about the control settings for local retrieval. Section 4 describes controlling for the reference element pair, but Section 5.2 only fixes the elemental identity of one endpoint, leaving the other endpoint as a potential cue for distinguishing candidates. In addition, the Gaussian radial reference uses unit-weight atom-centered Gaussians without explicitly encoding elemental species. Therefore, the learned models may exploit elemental identity information that is not explicitly encoded by this reference. It remains unclear how much of the performance improvement reported in Table 4 comes from fine-grained local environment modeling. This raises concerns about the reliability of this task for comparing local representation quality.
  2. EquiformerV2, CHGNet, and MACE are relatively dated models. I suggest extending the evaluation to more recent SOTA models (e.g. EquiformerV3, MatRIS, eSEN, and PET).
Questions for authors
  1. In the Joint setting, how many negative candidates share the same pair of endpoint elements as the query? How many candidates can be eliminated based solely on the elemental identity of the other endpoint?
  2. If the elemental identities of both endpoints are fixed, do the retrieval performance and encoder rankings in Table 4 remain consistent?
  3. See weakness 2, could the authors provide evaluation results for more recent models?
12 comments
AuthorsOpenReview ↗

Composition of Joint candidates (Weakness 1; Question 1)

We thank the reviewer for recognizing the value of HEABench and its magnetic-transfer results. We have completed the additional endpoint controls, element-weighted Gaussian references, and evaluations of all four suggested newer backbones.

In the original Joint setting, fewer than one of the three negatives per query shares both endpoint elements on average, while just over two can be excluded using the other endpoint’s element alone (0.827 and 2.173, respectively, across 2,777 queries). We have therefore added a control fixing both endpoint elements. We would also clarify throughout the manuscript that the original Joint setting fixes the parent and one endpoint element.

AuthorsOpenReview ↗

Retrieval with both endpoints fixed (Weakness 1; Question 2)

The encoder rankings remain highly consistent after both endpoint elements are fixed, although some absolute retrieval scores decrease. EquiformerV2+LSHCE retains 100% R@1 and MRR under the stricter control. On the same 416 queries, the Spearman rank correlation is 0.9912 for R@1, with a paired parent-bootstrap 95% interval of [0.9617,0.9948][0.9617,0.9948], and 0.980 for MRR. EquiformerV2+LSHCE remains at 100% R@1 and MRR. Some absolute scores decrease under the stricter control, but all combinations remain above the 25% chance level, with R@1 ranging from 61.78% to 100%. These results support the original conclusion that T2 distinguishes local representations beyond parent and endpoint identity.

Table below compares all 16 encoder combinations on the same 416 queries. Scores are averages of the 2 retrieval directions, reported as percentages.

\text{Encoder pair}&\text{Original Joint R@1}&\text{Both-endpoint R@1}&\text{Both-endpoint MRR}\\\\ \hline \text{EquiformerV2+PointVector}&97.72&94.71&97.26\\\\ \text{EquiformerV2+X-3D}&97.84&95.55&97.76\\\\ \text{EquiformerV2+PTv3-HEA}&94.95&94.95&97.41\\\\ \text{EquiformerV2+LSHCE}&100&100&100\\\\ \text{GeoFormer+PointVector}&87.74&81.85&90.28\\\\ \text{GeoFormer+X-3D}&85.22&79.09&88.49\\\\ \text{GeoFormer+PTv3-HEA}&88.22&85.1&92.07\\\\ \text{GeoFormer+LSHCE}&99.04&98.44&99.06\\\\ \text{CHGNet+PointVector}&81.25&76.32&86.73\\\\ \text{CHGNet+X-3D}&80.29&73.44&85.19\\\\ \text{CHGNet+PTv3-HEA}&78.97&75.84&86.73\\\\ \text{CHGNet+LSHCE}&89.78&83.17&90.91\\\\ \text{MACE+PointVector}&76.92&71.15&83.8\\\\ \text{MACE+X-3D}&74.52&67.31&81.32\\\\ \text{MACE+PTv3-HEA}&70.31&65.62&80.41\\\\ \text{MACE+LSHCE}&70.07&61.78&78.06\\\\ \end{array}$$
AuthorsOpenReview ↗

Elemental weighting in the Gaussian reference (Weakness 1)

Adding elemental weights strengthens the Gaussian reference, while the leading learned models still outperform it. PAW valence weights raise original Joint R@1 from 77.93% to 85.74%. On the 416-query both-endpoint task, this stronger reference reaches 88.34%, below five of the original 16 and 16 of the expanded 32 learned combinations. EquiformerV2+LSHCE reaches 100% under the same condition. Thus, the advantage of the leading models persists after elemental weights are included in the reference.

Tables below shows the results under fixed field width, radial pooling, normalization, and candidate lists, while the radial weights change across yje three references.

\text{Radial weights}&\text{Original Joint (2,777 queries)}&\text{Joint (416 common queries)}&\text{Both endpoints (416 common queries)}\\\\ \hline \text{Unit weights}&77.93/87.48&80.29/88.69&83.29/90.48\\\\ \text{Atomic number}&78.30/87.74&78.61/87.78&81.13/89.25\\\\ \text{PAW valence weights}&85.74/92.10&87.02/92.71&88.34/93.45\\\\ \end{array}$$

Comments on the paper

2 comments

We sincerely thank all reviewers for their constructive feedback and for recognizing the value of paired structural and electronic data for high-entropy alloys. Throughout the rebuttal period, we have conducted additional experiments, strengthened evaluation controls, clarified methodological details, and expanded the materials available for review. We would like to summarize these updates and the current status of HEABench.

Main Contribution

HEABench provides paired structural and electronic data with controlled evaluation for chemically disordered alloys. It contains 3,109 quinary, 100-atom configurations spanning 60 elements, with magnetic labels for a subset. Its three tasks, including global property prediction, local structure–charge matching, and low-label magnetic transfer, support complementary studies of predictive utility, local correspondence, and representation transfer. Together, these data and protocols provide a shared basis for investigating how local chemical environments relate to electronic behavior and material properties.

Repository and Review Materials

Repository: Anonymous HEABench repository

The repository now provides code, fixed data splits, running instructions, and three complete sample packages for review. These packages include original VASP outputs (atomic structures, CHGCAR charge-density files, DOS, and eigenvalues). The sample packages and datasets share the same packaging structure. A lightweight HTML gallery also provides visual examples of relaxed structures. Together, these materials allow reviewers to inspect the data format, representative records, and benchmark workflow.

Additional Experiments and Clarifications

  1. Expanded prediction and transfer baselines. We added zero-shot, fine-tuning, and scratch comparisons for MACE and CHGNet. Fine-tuning reduces mean MAE relative to scratch by approximately 42% and 18%, respectively, with improvements in all three paired runs. Adding EquiformerV3, MatRIS, eSEN, and PET also expands local matching to 32 encoder combinations.

  2. Repeated global prediction with complete charge grids. Across three runs with matched grids, encoders, and training settings, real charge density yields lower mean MAE and RMSE than the Gaussian control on all seven targets. Formation energy and both count targets improve on both metrics in every run.

  3. Stronger magnetic-transfer evidence. Across three downstream training runs, charge pretraining consistently reduces site MAE relative to scratch at 300 and 620 labels. Using 300 labels approaches the accuracy of scratch training with 620 labels. At 50 labels, improvements over Gaussian pretraining are consistent for low-moment sites, while overall gains vary across runs.

  4. Clarified physical settings and data overlap. We clarified NSP/SP calculations, structural initialization, and DFT settings. The NSP pretraining pool and SP test set contain no identical atomic structures or stoichiometries, although some five-element sets overlap. Magnetic prediction uses structures alone at inference.

  5. Refined interpretation of charge-density benefits. The learned three-dimensional Gaussian control highlights the strong contribution of geometry to retrieval. The matched prediction and transfer experiments support the additional value of charge density; isolating electronic rearrangement through retrieval remains an open question.

  6. Stronger local retrieval controls. Fixing the parent configuration and both endpoint elements preserves the original rankings closely (Spearman correlation: 0.9912). We added element-weighted Gaussian references and harder negatives. On 2,184 shared queries from 244 parents, geometrically hard negatives distinguish previously saturated strong models, with eSEN’s paired R@1 improvements over EquiformerV2, EquiformerV3, and MatRIS having confidence intervals that exclude zero.

Concluding Remarks

Overall, these updates provide broader baseline coverage, stronger experimental controls, and additional evidence for the utility of charge-density information, while clarifying the scope and limitations of our conclusions. We believe they have substantially strengthened the benchmark and its evaluation.

We sincerely appreciate the reviewers' time and continued engagement. We remain open to further discussion and are fully available to address any additional questions, concerns, or suggestions.

AuthorsOpenReview ↗

Author summary of the discussion period

Thanks again to all reviewers for their thoughtful feedback and constructive discussion, which have helped strengthen HEABench and its evaluation.

We will openly release the complete HEABench dataset, code, fixed data splits, and supporting materials to support reproducible research.

HEABench pairs structural and electronic data for 3,109 quinary, 100-atom configurations spanning 60 elements, with magnetic labels for a subset. Its three tasks cover global property prediction, local structure–charge matching, and low-label magnetic transfer.

The main additions during the rebuttal were:

  • Broader baseline coverage. Added pretrained-model evaluations, fine-tuning comparisons, and more recent encoders to provide stronger reference results.
  • Stronger prediction and transfer evidence. Added matched Gaussian controls and repeated experiments, strengthening evidence for charge density’s predictive and transfer benefits.
  • More discriminative retrieval evaluation. Expanded candidate sets and introduced geometrically hard negatives, distinguishing previously saturated strong encoders while preserving strict element-identity controls.
  • Clearer methodology and interpretation. Clarified physical calculation settings, data separation, and the respective contributions of geometry and charge-density information.

We hope HEABench will provide a useful foundation for multimodal representation learning and a deeper understanding of high-entropy alloys.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.