acceptodds
Under review as a conference paper at ICLR 2027

Average Accuracy Does Not Certify Topology in Learned Hamiltonians

Abstract

Learned electronic Hamiltonians are commonly evaluated by errors in matrix elements, band energies, or occupied subspaces, but when do these metrics certify topology? We show that no strictly positive finite-\(p\) integrated-error threshold universally certifies topology over unrestricted smooth gapped families: a topology-changing discrepancy can be confined to arbitrarily small support while retaining finite pointwise amplitude. Conversely, topologically distinct, well-defined rank-\(r\) occupied bundles must attain subspace infidelity at least \(1/r\) somewhere, motivating uniform, gap-aware control. Learned four-band models realize this mismatch: lower band MAE ranks a family with \(0/15\) correct topologies ahead of one with \(15/15\) correct. In realistic settings, a structured perturbation changes the Chern number of a \(32\)-orbital NaRuO Wannier Hamiltonian at \(8.074\) meV near-Fermi band MAE, while a released neural Hamiltonian undergoes gap collapse that invalidates a raw topological comparison. Equal-budget loop-structured sampling shows no stable advantage. In an exploratory \(10\)-meV band-matched study, all \(276\) evaluations valid at both endpoints preserve topology, and no target-valid plane becomes newly unsafe. Together, these results expose a certification gap between predictive accuracy and topological fidelity without implying that such failures are prevalent under generic small spectral perturbations.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
learned electronic Hamiltonians, topological fidelity, topology certification, electronic structure, topological materials, occupied subspaces, Chern and Z2 invariants

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RTLqOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether, when the prediction error of a learned electronic Hamiltonian is small, the topological correctness of the subspace it occupies can be guaranteed. The authors construct smooth, bandgap-bearing examples in which topological discrepancies are concentrated within arbitrarily small regions, allowing the error in finite p-order integrals to be arbitrarily small. On the other hand, the paper links topological inconsistency to the consistency distance of the projection operators, and provides sufficient guarantees based on the consistency projection distance or the Hamiltonian error relative to a reference energy gap. Finite-point sampling requires additional coverage conditions and independently proven regularity bounds.

Controlled four-band learning experiments show that the band MAE may rank a set of topologically incorrect models ahead of correct ones. Real-World Hamiltonian Cases The paper distinguishes between topological sensitivity caused by structured perturbations and the loss of numerical or spectral validity. It also reports negative results: cyclic finite-point samplin shows no stable advantage within the same budget, and in exploratory 10 meV material studies, all 276 evaluations with validity at both ends preserved topological properties. Therefore, the contribution lies in distinguishing between average accuracy and strict guarantees, rather than estimating the prevalence of errors in models used in actual deployments.

Critical strengths
  1. The paper clearly establishes a connection between the evaluation criteria and the downstream properties for which no guarantees can be provided. The localization construction intuitively demonstrates the limitations of finite-order errors, while the consistency conditions explain which stronger information is sufficient to provide guarantees. The paper distinguishes between families of smooth functions subject to no additional uniform constraints and cases with verifiable regularity bounds, while also acknowledging that positive stability tools stem from classical results.

  2. The scope of the empirical conclusions is defined with considerable rigor. The paper distinguishes between valid topological inconsistencies and invalid invariant computations, performs numerical resolution checks, and reports both positive examples and null results. In particular, the authors retain the results from the 10 meV study that revealed no topological inconsistencies, as well as the results showing that ring sampling lacks stability advantages, without using these to imply general failure or a universal solution. These qualifications significantly enhance the credibility of the interpretation of the results.

Critical weaknesses
  1. The failure labels in the evaluation benchmark need to be explained more clearly in the main text. Section 5 reports 75 failures out of 216 instances, while Appendix I.5 explains that the benchmark uses composite failure labels, which differ from inconsistencies with valid topologies. This distinction is important because the projection witness results do not apply to all bandgaps or numerical validity failures.

  2. There is one clear editorial issue in the manuscript. The Ethics Statement on page 10 describes the distribution of language model responses, pre-trained language models, and registered representation sets, which is inconsistent with this Hamiltonian study and should be replaced with a statement that accurately describes this research. This affects confidence in the quality of the final review of the manuscript.

Questions for authors
  1. Existing Failure Labels (Request for Clarification): Please specify in Section 5 which scenarios are covered by the composite failure label, and, based on the existing labels, briefly describe the proportion of the 75 failures attributable to valid topology inconsistencies versus validity-related failures. Please maintain a distinction between these and the conditions under which the projection witness results apply.

  2. Numerical Validity (Request for Additional Explanation): The validity rules are provided in Appendix I.5. Could you add an explicit reference in Section 6 to clarify that 0.25 meV is an operational choice specific to the numerical settings used, which must be combined with convergence and conditional checks, rather than a universal physical threshold?

  3. Ethical Statement (request for revision of the manuscript): Please replace the statement on page 10 regarding language model responses and representation groups with a statement describing the Hamiltonian data and research methods actually used in this paper.

5 comments
AuthorsOpenReview ↗

Response to Reviewer RTLq (1/3)

Thank you for the careful and constructive review. You found that the paper "clearly establishes a connection between the evaluation criteria and the downstream properties for which no guarantees can be provided", and you wrote: "The localization construction intuitively demonstrates the limitations of finite-order errors" and "The scope of the empirical conclusions is defined with considerable rigor." Your request to separate valid topological mismatches from validity failures led to a full validity audit of the benchmark (Sec. 5; App. J.3). We kept, in the main text, the scope statements and null results you valued.

Your commentShort answerWhere in the revised PDF
Q1 / W1. Which scenarios the "composite" failure label covers; what share of the 75 failures are valid mismatches vs. validity failures; keep the distinction from where the projector witness appliesThe labels are family-specific Chern-label disagreements, not composite. 69 of the 75 are valid mismatches, 0 are invalid predictions, 6 are unresolved. Sec. 5 states that the witness applies to the valid mismatchesSec. 5, ll. 304–323; App. I.5, J.1, J.3; Tables 21, 24
Q2. State in Sec. 6 that 0.25 meV is an operational choice combined with convergence and conditioning checksDone in Sec. 6 and Fig. 4, matching App. I.5; a stricter gap-clearance audit is addedll. 341–344, 458; App. I.5, J.11
Q3 / W2. Replace the Ethics StatementReplacedll. 548–552

Q1 / W1. Please specify in Sec. 5 which scenarios the composite failure label covers, give the proportion of the 75 failures attributable to valid topological mismatches versus validity failures, and keep these distinct from the conditions of the projector witness.

Short answer. Done. Tracing the frozen benchmark code showed that the 75 labels are family-specific Chern-label disagreements. Gap, link and convergence failures were never folded in, so "composite" was the wrong word. A validity audit of both endpoints finds that 69 of the 75 are valid topological mismatches, none is an invalid prediction, and 6 are unresolved.

Details. Label rules and composition (App. J.1, Table 21):

FamilyFailure ruleInstancesFailures
Two-band exact∣ΔC∣>0.25\lvert\Delta C\rvert>0.2555
Two-band learnedstored topology-correct flag (tolerance 0.20)7235
Two-band rescuestored topology-correct flag436
Four-band exact∣ΔC∣>0.25\lvert\Delta C\rvert>0.2555
Four-band learnedstored topology-correct flag3815
NaRuO₂ structureddeviation from C=−2C=-2 above 0.25139
NaRuO₂ randomdeviation from C=−2C=-2 above 0.25400
Total21675

Validity gates, fixed before evaluation and applied at both endpoints:

  • link singular value above 10−410^{-4} (two-band solid-angle instances: triangle margin);
  • sampled gap above 1% of the reference gap;
  • the same label on the two finest recorded grids and all retained origins.

An endpoint recorded on a single grid counts as unresolved, not as passing. Results by family (Table 24):

FamilyFailures: valid mismatch / unresolvedNon-failures: valid agreement / valid mismatch / unresolved
Two-band exact0 / 5—
Two-band learned35 / 029 / 1 / 7
Two-band rescue5 / 132 / 0 / 5
Four-band exact5 / 0—
Four-band learned15 / 023 / 0 / 0
NaRuO₂ structured9 / 04 / 0 / 0
NaRuO₂ random—40 / 0 / 0
Total69 / 6128 / 1 / 12

No instance is an invalid prediction. Five of the six unresolved failures are the exact two-band constructions of Sec. 3: they were recorded on a single grid, and their Chern difference is known analytically. Current status over all 216: 70 valid mismatches, 128 valid agreements, 18 unresolved.

AuthorsOpenReview ↗

Response to Reviewer RTLq (2/3)

Q1 / W1 (continued). The benchmark conclusion is unchanged under either labeling (App. J.1, J.3):

InstancesLabelsAUROC loop / stratified IID, B=64B=64Difference [95% family-cluster interval]One-sided pp
All 216historical0.8445 / 0.8456−0.0010 [−0.039, 0.042]0.51
198 valid at both endpointscurrent status0.8470 / 0.8415+0.0055 [−0.031, 0.056]0.36
198 valid at both endpointshistorical0.8486 / 0.8450+0.0036 [−0.034, 0.056]0.42

Changes in the paper. Sec. 5, ll. 308–315:

... 75 of which are labeled failures by family-specific Chern-label disagreement rules. The labels do not fold in gap or conditioning failures. [...] 69 of the 75 failures are valid mismatches and none is an invalid prediction; the other six are unresolved, five of them the exact constructions of Sec. 3, recorded on a single grid (Appendix J.3). The witness of Proposition 2 applies to the valid mismatches.

App. I.5 (ll. 2793–2798) keeps the distinction you asked for:

This taxonomy is distinct from the historical family-specific topology-label disagreement used in the finite-query benchmark [...]. The pointwise witness theorem applies to two continuous, equal-rank valid bundles with different topology; it cannot automatically be applied to all 75 recorded label failures without those assumptions.

Q2. Could you add an explicit reference in Sec. 6 clarifying that 0.25 meV is an operational choice specific to the numerical settings, combined with convergence and conditioning checks, rather than a universal physical threshold?

Short answer. Done, in Sec. 6, in the Fig. 4 caption, and consistently with App. I.5. We also audited every stored Wilson flow with a stricter, Z2Pack-style gap-clearance criterion (App. J.11).

Details. The guardrail is one of four joint checks: gap above 0.25 meV, Kramers error below 0.01, Wilson-link singular value above 10−410^{-4}, and agreement of the two finest Wilson-loop resolutions. Key outcomes of the clearance audit (criterion: clearance at least 0.3; every case is listed in App. J.11):

CaseGap-clearance outcome
ZrSiPt, NaCaTaTiO₆all planes pass at both endpoints
Ta₃AlC₂the four kxk_x, kyk_y planes pass; the adaptive kzk_z flows pass
Ca₃Si₄Ir₄all planes fail, consistent with its invalid classification
Strained BiTeI, s=0.91s=0.91, 0.920.92all 144 dense flows pass (minimum 0.42 / 0.49)
GeBi₄Te₇ mismatch records (6)both endpoints dense-verified; minimum prediction clearance 0.497

The audit is reported in full, including the cases where a stored flow does not by itself establish its label under this stricter criterion; the stress-study counts keep that study's three-grid protocol of record, with the dense statuses reported alongside (App. J.11).

Changes in the paper. Sec. 6, ll. 341–344:

[...] with Z2Z_2 on the six time-reversal-invariant planes and a validity protocol: an operational 0.25-meV gap guardrail plus Kramers, link and two-finest-grid checks (Appendix I.5; Table 2).

Fig. 4 caption (l. 458):

the 0.25-meV line is the operational validity guardrail, not a phase boundary.

App. I.5 (ll. 2789–2792):

The exact 0.25-meV cutoff is an operational, conservative heuristic for the frozen grids. It is neither a universal validity threshold nor a physical phase boundary; the diagnosis is supported jointly by gap collapse, convergence behavior, Kramers/link checks, and resolution stability.

AuthorsOpenReview ↗

Response to Reviewer RTLq (3/3)

Q3 / W2. Please replace the Ethics Statement on page 10 with a statement describing the Hamiltonian data and methods actually used.

Short answer. Thank you for catching this; it is replaced.

Changes in the paper. Ethics Statement, ll. 548–552:

This work studies computational electronic Hamiltonians using controlled models, released electronic-structure data and released predictor artifacts. It involves no human participants or personal data. Data and software sources are identified, and no third-party model weights or raw datasets are redistributed.

You noted that this error affected confidence in the final review of the manuscript. For this revision we therefore re-verified every number in the submission against its source data. The resulting corrections are listed in full in the general response (part 2/3). Table 20 now gives, for each empirical result, its evidence, validity protocol and status.

The scope statements and null results you valued are kept.

Kept in the revisionWhere
The 10-meV study: 276 evaluations valid at both endpoints, 0 valid mismatches, 0 newly unsafe. Now extended to 20 meV, with still no valid mismatchSec. 7, ll. 424–426; App. H.3, Tables 17–18
Loop-structured queries show no stable detection advantage at equal costSec. 5, ll. 322–323; App. F, Table 13
New: the same holds design by designSec. 5, ll. 316–317; App. F.3
The stress counts locate vulnerable planes rather than estimate prevalenceSec. 7, ll. 461–462
All realistic-model controls reported in fullApp. E.2, J.4, J.7

The equal-cost AUROC across budgets is unchanged (Table 13):

Budget BBLoop-structuredIIDStratified IIDRandom paths
160.76470.73510.70960.7068
320.86630.77880.83980.8519
640.84450.84210.84560.8778
1280.88800.90080.87890.8624

Scored design by design (16 designs at B=64B=64), the mean AUROC is 0.589 (loop), 0.650 (IID), 0.647 (stratified IID) and 0.682 (random path). The loop-minus-stratified difference is positive in 4 of 16 designs on the 216 instances, and in 4 of 16 on the 198 valid at both endpoints (App. F.3).

The general response summarizes the evidence added for the other reviewers, including a released predictor that returns a converged wrong Z2Z_2 class on strained BiTeI (Sec. 6, Table 3).

If these changes address your questions, we would be grateful if you would consider raising your score.

Reviewer_3hZWOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies ML-based methods to generate Hamiltonian matrices, mathematical constructs that desctibe a material's electrons. The authors main claim is that the l-p (p is finite) reconstruction error of the matrix is not a reliable indicator of the quality of the generated hamiltonian since underlying properties such as topology are not bounded by average error. In other words, an ML-estimated Hamiltonian with arelatively small l-p (with finite p) reconstruction error may yield a topology that significantly differs from the material's.

Critical strengths

The paper has a smooth writing and methodology, it states the finding in clear terms and performs experiments to support them.

Critical weaknesses

I do not have the expertise to review this paper, I will defer the decision to other reviewers with higher expertise level on the subject.

Questions for authors

My reading from the paper is that l-p reconstruction bounds do not guarantee reconstruction quality. if that is the case:

  • is there a calibrated model that can generate the reconstruction with an estimated "certainty" on the reconstruction correctness?
  • would it help to train models with l-inf objectives on reconstruction error? are such models trained and studied in the literature?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 3hZW (1/2)

Thank you for the review. You found that the paper "has a smooth writing and methodology, it states the finding in clear terms and performs experiments to support them". Your two questions, on calibrated certainty and on L∞L^\infty objectives, led to two new experiments: 60 models trained with power-mean (max-error) losses, and six-member ensembles for all 30 four-band configurations (Sec. 9; App. J.8, J.9).

Your commentShort answerWhere in the revised PDF
Reading: LpL^p bounds do not guarantee reconstruction qualityFor topology, yes: an LpL^p bound guarantees LpL^p accuracy but not the topological class; uniform control doesSec. 4; Sec. 9, ll. 525–526
Q1. Is there a calibrated model that gives a certainty of correctness?Calibration is possible relative to a deployment distribution, not as a universal certificate. Ensemble votes are unanimous and wrong; a uniform disagreement between members separates the cohorts in medianSec. 9, ll. 525–534; App. J.9
Q2. Would L∞L^\infty objectives help? Are they studied?Tested: power-mean training with p=8p=8 halves the training maximum but barely moves the dense sampled maximum (1.90 → 1.89); neither p=8p=8 nor p=64p=64 changes any label. To our knowledge, no Hamiltonian-learning method trains on a continuous uniform errorSec. 9, ll. 527–539; App. J.8, B.2, D.3

Your reading: LpL^p reconstruction bounds do not guarantee reconstruction quality.

Short answer. For topology, yes. An LpL^p bound guarantees accuracy in LpL^p, but over unrestricted smooth families it cannot guarantee the topological class. Uniform control can.

Details.

Error controlledDoes it fix the topological class?Where
LpL^p error, any finite ppNo positive threshold sufficesProp. 1
Band energies, even exactly at every kkNo: in Sec. 3 the band energies coincide exactly while the Chern numbers differSec. 3, ll. 137–142
Agreement on a fixed finite point setNo, without regularityProp. 4
Uniform Hamiltonian error below half the gapYesProp. 3
Uniform projector distance below oneYesProp. 2

Q1. Is there a calibrated model that can generate the reconstruction with an estimated "certainty" on its correctness?

Short answer. Not as a universal certificate: no average error can provide one (Prop. 1). A calibrated probability of correctness is well defined relative to a deployment distribution, and the revision tests what such an estimator can rely on. A widely used proxy for certainty, the ensemble vote, is unanimous and wrong in all 15 IID configurations. A reference-free uniform quantity, the largest projector distance between members, separates the two cohorts in median (0.40 vs. 0.16).

Details. For each of the 30 four-band configurations of Table 1 we trained five additional members that differ only in their initialization seed, and took the six-member vote on the Chern label (App. J.9):

CohortEnsemblesVoteMajority correctMedian max. pairwise member projector distance
IID15unanimous in 15/150/150.40
Oracle loops15unanimous in 15/1515/150.16
  • Why the vote fails. Members trained on the same points share what those points miss, so the vote fraction is 1 for every wrong majority. It is not an estimated probability of correctness.
  • What a calibrated certainty needs.
    1. Fix the correctness event: valid agreement of the label at both endpoints.
    2. Fix a deployment distribution.
    3. Estimate the probability of the event on held-out references from that distribution, using uniform diagnostics such as member disagreement as inputs.
  • Where certainty is deterministic. With a reference, the uniform conditions give certificates (Sec. 4).

Our benchmark AUROC ranks instances; it does not calibrate them.

Changes in the paper. Sec. 9, ll. 530–534:

Six-member ensembles trained on the same points vote unanimously for the wrong label in all 15 configurations, whereas the largest projector distance between their members is 0.40 (median) against 0.16 for the correct cohort [...]. A calibrated probability of valid agreement is well defined relative to a deployment distribution; the AUROC of Sec. 5 measures ranking, not calibration.

AuthorsOpenReview ↗

Response to Reviewer 3hZW (2/2)

Q2. Would it help to train models with L∞L^\infty objectives on the reconstruction error? Are such models trained and studied in the literature?

Short answer. We tested the closest trainable version: power-mean losses that approach the maximum over training points. Without a regularity bound, a maximum over training points does not control the continuous maximum, and in our test it changes no label. To our knowledge, no Hamiltonian-learning method trains on a continuous uniform error.

Details. We kept the data, seeds, architecture, optimizer and 2200-step budget of Table 1 and replaced only the MSE loss by the power mean (1n∑ieip)1/p\big(\tfrac1n\sum_i e_i^p\big)^{1/p} of the per-point errors, for p=8p=8 and p=64p=64 (60 new models). As pp grows, this approaches the maximum over training points. Cohort medians on 96296^2 (App. J.8, Table 30):

LossCohortCorrectBand MAEMean ffMax ffTrain maxDense max
MSEIID0/150.00280.00120.4970.0271.90
MSEloops15/150.00621.1×10−51.1\times10^{-5}0.00110.0620.145
p=8p=8IID0/150.00430.00120.4890.0141.89
p=8p=8loops15/150.00914.3×10−54.3\times10^{-5}0.00150.0530.140
p=64p=64IID0/150.01390.00130.4880.0411.89
p=64p=64loops15/150.02694.7×10−44.7\times10^{-4}0.00370.0870.227
  • No label changes. IID stays 0/15 and loops 15/15, on both 96296^2 and 1282128^2.
  • The training maximum halves with p=8p=8; the dense maximum barely moves. The median largest training-point error falls from 0.027 to 0.014, while the sampled maximum operator error on 96296^2 moves only from 1.90 to 1.89, and the sampled maximum infidelity stays near 0.49, close to the witness value 1/r=1/21/r=1/2. With p=64p=64 and the same step budget, the training maximum does not fall (0.041).
  • Why. Between training points the error is unconstrained, which is where the topological texture lies. A maximum over training points bounds the continuous maximum only with a regularity bound (Sec. 4.4, Eq. (4)).
  • Where an integrated objective does suffice. For errors with MM lattice-vector components, ∥E∥∞≤M∥E∥L2,F\Vert E\Vert_\infty\le\sqrt{M}\Vert E\Vert_{L^2,F}, so ∥E∥L2,F<Δ∗/(2M)\Vert E\Vert_{L^2,F}<\Delta_*/(2\sqrt M) certifies (App. B.2).

Literature.

ApproachWhat it controls
Equivariant Hamiltonian-learning methodsevaluated by average errors (ll. 47–50)
WALoss (Li et al., 2025)eigenspace alignment, on average
NextHAM (Yin et al., 2026; ICLR 2026)reciprocal-space and subspace supervision, on average
Power-mean losses (this revision)the maximum over training points; no label changes
Dirichlet regularization (App. D.3)smoothness of the extension: recovery 12/15 → 14/15, median Dirichlet energy 61.81 → 19.73

The Dirichlet result suggests inductive biases toward smoother extensions as a route worth studying, because they act between training points, where a training-point maximum cannot.

Changes in the paper. Sec. 9, ll. 525–539, new paragraph "Calibration and uniform training", which includes:

To our knowledge, no Hamiltonian-learning method trains on a continuous uniform error: the closest objectives, WALoss and NextHAM, supervise eigenspaces and subspaces on average (Li et al., 2025; Yin et al., 2026); our Dirichlet regularization raises recovery from 12/15 to 14/15 as an inductive bias (Appendix D.3).

For readability, we also rewrote the introduction and revised Secs. 3 and 5–7; the contributions are listed at ll. 92–107.

If these answers address your questions, we would be grateful if you would consider raising your score.

Reviewer_3hZWOpenReview ↗

keeping score unchanged

I thank the authors for their responses. I only had minor comments which the authors have thoroughly responded to.

For fairness, I keep my score the same as a borderline accept decision and defer the actual rating to other reviewers with better aligned background.

We appreciate you taking the time to read our responses. We understand and respect your decision. Your two questions led to two new experiments in the revision, on max-error training and on ensembles (App. J.8, J.9).

Reviewer_Aa8qOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The submission highlights a potential gap between accuracy and topology in electronic Hamiltonians. Specifically, it's possible to localize a discrete topological deviation to a region of vanishing extent such that the averaged difference between a Hamiltonian with and without the deviation also vanishes. This might have ramifications for a practitioner hoping to capture topological structure while only benchmarking by eg MAE. Multiple empirical characterizations examine the effect from different angles; in one, the authors optimize two Hamiltonians on one of the ~pathological constructions which either sample broadly or densely around the topological perturbation. For some metrics, the model that correctly captures the topology will be the one with worse performance. Thus solely looking at accuracy might hide a failure to capture topology.

Critical strengths

Characterizing the relationships between relevant metrics in ML is worthwhile both practically (can I use a particular error as a sufficient statistic for all others?) and for a deeper understanding of the nature of optimization. Here, the relationship between a global measurement (topology) and averaged local agreement is illuminated through a balance of constructed scenarios, formal analysis, and empirical tests.

Critical weaknesses

The central failure mechanism -- of the possible disconnect between topology and locally evaluated mismatch -- is not demonstrated to have any real import. The paper follows the arc of a manufactured conflict followed by a mild degree of resolution. There is insufficient evidence that this is a problem in practice -- that nontrivial topology stashed below the sampling resolution afflicts Hamiltonian optimization -- nor is it much more surprising than the well-worn ML phenomenon where optimizing one quantity need not determine performance with respect to another.

In the headline result of the abstract and fig 1 ("Learned four-band models realize this mismatch: lower band MAE ranks a family with 0/15 correct topologies ahead of one with 15/15 correct." L18), the result is much weaker than presented. On Hamiltonian RMSE and the mean and max subspace infidelity, the Hamiltonian with correct topology is also better by performance on these metrics. It appears from the code that RMSE was used for training; band MAE is the only metric pointing toward the wrong topology and it's distinct from what was optimized.

Note also that the ethics statement seems to pertain to a different paper.

Questions for authors

For the constructed examples where the topological structure has vanishing extent -- can the authors show any cases where actual Hamiltonian predictors exhibit anything similar? The paper needs to show practical relevance rather than possibility in constructed scenarios.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Aa8q (1/5)

Thank you for the careful review. You found that characterizing the relationships between metrics is "worthwhile both practically (can I use a particular error as a sufficient statistic for all others?) and for a deeper understanding of the nature of optimization", and that the relation between topology and averaged agreement "is illuminated through a balance of constructed scenarios, formal analysis, and empirical tests". Your question, whether actual Hamiltonian predictors exhibit anything similar, shaped this revision: we ran a released predictor on a real material and found a converged wrong topological class (Sec. 6, Table 3). We number your points as below and start with Q.

Your commentShort answerWhere in the revised PDF
Q. Can actual Hamiltonian predictors exhibit anything similar?Yes. A released Uni-HamGNN predictor assigns strained BiTeI a converged wrong Z2Z_2 class; in the released audit, the prediction with the lowest band MAE is the only invalid oneSec. 6, ll. 341–393; Tables 2–3; App. J.10, J.11
W1a. No demonstrated real import; "manufactured conflict"Tested in three settings, each addressing a different part of the mechanism: a released predictor, a 32-orbital Wannier model with 160 controls, and 24 JARVIS materialsSecs. 6–7; App. E.2, H.3, J.5, J.7
W1b. Not more surprising than optimizing one quantity not determining anotherThe paper identifies which quantity certifies topology and what value a class change requires: no finite-pp threshold exists, while the uniform threshold is sharpSec. 4, ll. 202–269; App. B.2, J.5
W2. Four-band headline weaker than presented; band MAE is the only metric pointing wrongAgreed; now stated in the abstract, Fig. 1, Sec. 3 and main-text Table 1, with the objective named; two remedies testedll. 23–25, 69–73, 145–155; Table 1; App. J.8, J.9
W3. Ethics statement belongs to another paperReplacedll. 548–552

Q. Can the authors show cases where actual Hamiltonian predictors exhibit anything similar?

Short answer. Yes. Run unmodified on strained BiTeI, the released Uni-HamGNN predictor gives the occupied bands a converged strong Z2Z_2 class, ν0=1\nu_0=1, while our DFT reference is trivial. The failure has the signature the paper predicts: on the disagreeing planes the mean subspace infidelity stays near 0.01 while the projector distance approaches one.

Details. BiTeI is not among the released examples. All references are our own OpenMX 3.9.9 PBE+SOC calculations in the released basis; the pipeline reproduces the released ZrSiPt reference Hamiltonian to 1.3×10−81.3\times10^{-8} eV and the released prediction up to a scalar multiple of the overlap. At 9% lattice compression (s=0.91s=0.91), plane by plane (Table 3; at s=0.92s=0.92 all labels are 0):

PlaneLabels ref/pred at s=0.91s=0.91Band MAE (meV), 0.92→0.910.92\to0.91Mean ff, 0.92→0.910.92\to0.91Max dd, 0.92→0.910.92\to0.91
kx=0k_x=0, ky=0k_y=00 / 1193 → 3150.0037 → 0.01170.583 → 0.9973
kz=12k_z=\tfrac120 / 1210 → 5640.0026 → 0.01020.580 → 0.99999
kz=0k_z=00 / 0153 → 1650.0011 → 0.00170.164 → 0.256
kx=12k_x=\tfrac12, ky=12k_y=\tfrac120 / 069 → 860.0005 → 0.00070.057 → 0.068

Both endpoints are valid, and both label sets are confirmed:

Check at s=0.91s=0.91ReferencePrediction
Direct gap, rank 38 (24324^3)0.13 eV0.30 eV
Gap, link and Kramers gatespasspass
Dense Wilson loops: 32 points, 129 and 257 slices, 3 originslabels keptlabels kept
Minimum gap clearance (criterion ≥ 0.3)0.490.42
Bi weight, conduction minus valence edge, at the reference gap minimum+0.050−0.033

On the disagreeing planes d>0.9d>0.9 on 6.1–9.2% of the plane; on the agreeing planes max dd stays at most 0.26 (App. J.10, Table 32).

Changes in the paper. Sec. 6 (ll. 368–393), new Table 3, App. J.10 (Tables 31–32); also the abstract (ll. 19–21) and introduction (ll. 87–90). Sec. 6 now reads (ll. 390–391):

Band MAE and mean infidelity measure how large the error is; the projector distance, read against its threshold of one, indicates whether the error can have changed the class.

AuthorsOpenReview ↗

Response to Reviewer Aa8q (2/5)

Q (continued). The mismatch is specific to one structure, and band MAE has no value that marks a class change.

Along two compression paths (isotropic ss; cc axis only, tt), the same predictor agrees with the reference at 15 of 17 structures (App. J.10, Table 31; gaps on 24324^3, band MAE on the 32232^2 kz=0k_z=0 grid). Labels are in the order kx=0,12k_x=0,\tfrac12; ky=0,12k_y=0,\tfrac12; kz=0,12k_z=0,\tfrac12, and the reference is 000000 at all 17 structures. Agree and valid mismatch mean that both endpoints pass every validity check, with equal or different labels; unresolved means that an invariant did not converge at an endpoint (Sec. 2, ll. 126–132; App. I.5):

StructureRef. gap (meV)Pred. gap (meV)Band MAE (meV)Pred. labelsClass
s=1.00s=1.00239.5261.846.0000000agree
s=0.99s=0.99241.0303.524.6000000agree
s=0.98s=0.98239.8283.035.5000000agree
s=0.97s=0.97236.1273.842.9000000agree
s=0.96s=0.96227.9324.338.5000000agree
s=0.95s=0.95215.8322.449.6000000agree
s=0.94s=0.94200.1332.881.6000000agree
s=0.93s=0.93179.5432.0126.5000000agree
s=0.92s=0.92154.0443.6153.0000000agree
s=0.91s=0.91125.7302.9164.9101001valid mismatch
t=0.97t=0.97218.3249.237.4000000agree
t=0.94t=0.94210.2253.634.9000000agree
t=0.91t=0.91217.3244.925.8000000agree
t=0.88t=0.88237.8239.031.8000000agree
t=0.85t=0.85254.6214.543.9000000agree
t=0.82t=0.82215.7214.478.9000000agree
t=0.79t=0.79145.7183.4153.3000001unresolved
  • Band MAE grows with strain and has no value at which the class must change. It is 126.5 and 153.0 meV at s=0.93s=0.93 and 0.92, where the class is correct, and 164.9 meV at the mismatch. On the three disagreeing planes it is already 193–210 meV at s=0.92s=0.92, again with the correct class (Table 3).
  • The projector distance marks the change. On those planes it is 0.58 at s=0.92s=0.92 and 0.997–0.99999 at s=0.91s=0.91, approaching the value a class change requires.
  • The predicted direct gap gives no warning. At the mismatch it is 302.9 meV, larger than the reference's 125.7 meV, and the predicted labels are converged. Compression is the regime in which pressure- or strain-tuned screening needs such a warning (App. J.10).

At t=0.79t=0.79 the kz=12k_z=\tfrac12 label differs between the two finest grids, so we count it as unresolved.

Q (continued). In the released audit, the most accurate prediction is the one whose topology cannot be compared.

All four complete released reference–prediction pairs, on a common 32232^2 grid (Table 2):

MaterialrrBand MAE (meV)Δ∗/2\Delta_*/2 (meV)Mean ffrmax⁡fr\max fMax ddSix-plane Z2Z_2
ZrSiPt3220.98972.3×10−42.3\times10^{-4}0.0100.0546/6 trivial, stable
NaCaTaTiO₆16017.411531.8×10−41.8\times10^{-4}0.0330.0426/6 trivial, stable
Ta₃AlC₂10020.035.43.4×10−43.4\times10^{-4}0.0770.1455/6 stable; kz=0k_z=0 unresolved
Ca₃Si₄Ir₄10611.10.776.1×10−46.1\times10^{-4}1.050.999997invalid on all planes
  • Ranked by band MAE alone, Ca₃Si₄Ir₄ would come first. Yet its kz=0k_z=0 gap collapses below the guardrail and it fails the validity gates on every plane.
  • Absolute band MAE does not identify the safest prediction. Relative to half the reference gap it is 0.02 for the two large-gap materials, 0.6 for Ta₃AlC₂ and 14 for Ca₃Si₄Ir₄.
  • The discrepancy is local and diluted by 1/r1/r. At one momentum, a predicted occupied direction is nearly orthogonal to the reference (max dd = 0.999997), while the maximum infidelity is only 0.0099≈1.05/r0.0099\approx1.05/r.
  • Adaptive refinement (App. J.6, Table 28). The region is small: d>0.5d>0.5 on 1.02% and d>0.9d>0.9 on 0.066% of the plane. The refined maximum is 1−10−91-10^{-9}, at a momentum where the reference local gap is 0.73 meV. The distance is already 0.72 on a 16216^2 grid and saturates under refinement.

Changes in the paper. Sec. 6, ll. 341–366, Table 2; App. G.2, J.2 and J.6. Sec. 6 concludes (ll. 362–363):

No topology claim, correct or incorrect, can therefore be drawn from the prediction with the lowest band MAE.

AuthorsOpenReview ↗

Response to Reviewer Aa8q (3/5)

W1a. The central failure mechanism is not demonstrated to have real import; the paper follows the arc of a manufactured conflict followed by a mild degree of resolution.

Short answer. The revision tests the mechanism in three settings, each addressing a different part of it: the actual error of a released predictor (Q above); direction versus size at matched norms in a realistic 32-orbital Wannier Hamiltonian; and where real JARVIS band structures are vulnerable under generic stress.

Details.

(i) NaRuO₂: matched norms, different outcomes (Sec. 6; App. E.2, J.7). The band-edge direction DD changes the Chern number, while 160 random settings at the same amplitudes keep C=−2C=-2 (Table 11; 2 amplitudes per direction, 64264^2):

DirectionRank∥D∥F\Vert D\Vert_F∥D∥op\Vert D\Vert_{\rm op}SettingsMin gap (meV)CC changes
Structured band-edge DD22\sqrt21215.2 / 6.72/2
Frobenius-matched random322\sqrt20.442–0.49340130.7–153.10/40
Operator-matched random322.871–3.199140108.6–157.90/40
Generic rank-two22\sqrt2140136.8–156.90/40
Rank two inside the band-edge span22\sqrt214016.0–215.20/40

A refined sweep along DD (Table 29) places the change within one 1-meV step of half the 144.8-meV sampled gap:

λ\lambda (eV)0.0700.0710.0720.0730.074
Refined gap (meV)4.772.770.771.223.22
Chern number−2−2−2−1−1

An interval enclosure (∥∇H∥≤596\Vert\nabla H\Vert\le596 eV, 2.9×1072.9\times10^7 cells) certifies that the reference gap never closes.

(ii) A small matrix error is not a small spectral error (Sec. 7). At a fixed real-space matrix MAE of 1.417 meV, three generic mechanisms across 168 JARVIS materials give median near-Fermi band MAEs of 601, 556 and 2262 meV.

(iii) Real band structures under stress (Sec. 7, Table 18). Valid mismatches on 24 JARVIS materials, by dose, under the study's three-grid protocol of record; each cell has 91–92 planes valid at both endpoints:

Dose (meV)102050100200
Uniform rescaling00000
Range attenuation00223
Paired-shell random00000

The first valid mismatches are two nontrivial planes of GeBi₄Te₇ (23-meV plane gap), confirmed by a dense re-evaluation with gap clearance (App. J.11).

(iv) Where the decisive momenta lie, on your point about topology "stashed below the sampling resolution" (Sec. 7; App. J.5, Table 27). By the pointwise certificate, a topology-changing error no larger than the minimum gap Δmin⁡\Delta_{\min} must reach half the local gap somewhere inside the region {g∗≤2Δmin⁡}\lbrace g_*\le2\Delta_{\min}\rbrace. This region locates where the error must act, and it is small on real planes:

PlaneΔmin⁡\Delta_{\min} (meV)Area of the region
GeBi₄Te₇ kx=0k_x=0 / ky=0k_y=023.20.169% / 0.171%
GeBi₄Te₇ kz=12k_z=\tfrac12124.30.449%
BaGaSnH kz=12k_z=\tfrac1259.50.025%
BaMg₂Bi₂ kz=0k_z=038.50.037%
Ta₃AlC₂ kx=0k_x=0 / ky=0k_y=051.53.99%

On ZnCu₂SnSe₄ ky=0k_y=0 (8 meV) and on Ta₃AlC₂, this region contains no point of a 12212^2 grid.

Changes in the paper. Sec. 6, ll. 401–416; Sec. 7, ll. 419–471. Sec. 7 ends (ll. 470–471):

The momenta that decide such a change can thus escape averages and coarse grids.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (1/3): strengths and new evidence

We thank the three reviewers for their careful reading. Section, table and line numbers refer to the revised PDF. Every new result has its data and scripts in the supplement's round2/ directory, and round2/MAP_LABELS.csv maps each table and figure to its files. Part 2/3 lists in full the corrections that came from the reviews and from our re-verification of the submission.

1. Strengths noted by the reviewers

  • The question. Relating evaluation metrics to downstream properties is "worthwhile both practically ... and for a deeper understanding" (Aa8q); the paper "clearly establishes a connection between the evaluation criteria and the downstream properties for which no guarantees can be provided" (RTLq).
  • Theory and experiment together. The relation between topology and averaged agreement "is illuminated through a balance of constructed scenarios, formal analysis, and empirical tests" (Aa8q); "The localization construction intuitively demonstrates the limitations of finite-order errors" (RTLq); the paper "states the finding in clear terms and performs experiments to support them" (3hZW).
  • Rigorous scope. "The scope of the empirical conclusions is defined with considerable rigor", and the retained null results "significantly enhance the credibility of the interpretation of the results" (RTLq).

2. New in the revision

New evidenceKey resultRelated commentWhere
Released Uni-HamGNN predictor, unmodified, on strained BiTeI, with our own OpenMX referencesConverged wrong class: reference trivial, prediction ν0=1\nu_0=1; on the 3 disagreeing planes mean ff = 0.010–0.012, max dd = 0.997–0.99999Aa8q QSec. 6, Table 3; App. J.10
All four released reference–prediction pairs on a common gridThe lowest band MAE (Ca₃Si₄Ir₄, 11.1 meV) is the only invalid prediction; max dd = 0.999997Aa8q QSec. 6, Table 2; App. J.2, J.6
NaRuO₂ (32 orbitals): band-edge direction vs. 160 random settings in four control groups (matched in Frobenius norm; in operator norm; in rank and both norms; in rank, both norms and support)CC changes between 72 and 73 meV, bracketing half the 144.8-meV gap; 0 of 160 controls change CCAa8q W1Sec. 6; App. E.2, J.7
Interval enclosure for NaRuO₂Reference gap certified never to close (∥∇H∥≤596\Vert\nabla H\Vert\le596 eV, 2.9×1072.9\times10^7 cells)Aa8q W1Sec. 4.4; App. J.7
24 JARVIS materials, band-matched stress at 10–200 meVNo valid mismatch at 10–20 meV; first valid mismatches at 50 meV, on small-gap nontrivial planesAa8q W1Sec. 7, Table 18; App. H.3
Gap-clearance audit of all stored Wilson flows; dense re-evaluationBiTeI labels and all GeBi₄Te₇ mismatch records confirmedRTLq Q2App. J.11
Where a topology-changing error must reach half the gap0.02–0.45% of the plane on the five dense-verified nontrivial planesAa8q W1Sec. 7, Table 27; App. J.5
Power-mean (max-error) training, p=8,64p=8, 64: 60 new modelsNo label changes (IID 0/15); dense max error 1.89 vs. 1.903hZW Q2, Aa8q W2Sec. 9; App. J.8, Table 30
Six-member ensembles for all 30 four-band configurationsUnanimously wrong in 15/15; median member distance 0.40 vs. 0.163hZW Q1, Aa8q W2Sec. 9; App. J.9
Validity split of the 216-instance benchmark75 failures = 69 valid mismatches + 0 invalid + 6 unresolvedRTLq Q1Sec. 5; App. J.3, Table 24
Single-design AUROCNo stable loop advantage design by design eitherRTLq (null results)App. F.3
Theory: finite-range bound; pointwise gap certificate∥E∥∞≤M∥E∥L2,F\Vert E\Vert_\infty\le\sqrt{M}\Vert E\Vert_{L^2,F}; a topology change needs ∥E(k)∥op≥g∗(k)/2\Vert E(k)\Vert_{\rm op}\ge g_*(k)/2 somewhereAa8q W1, 3hZW Q2Secs. 4.1, 4.3; App. B.2, J.5
Evidence ledgerEvidence, validity protocol and status of every empirical resultallTable 20
AuthorsOpenReview ↗

General response (2/3): corrections and changes in interpretation

3. Corrections and changes in interpretation

Rows 1–3 came from the reviews. The others came from our re-verification of every number in the submission against its source data, and from refinement under the final protocols. The theory and the direction of every empirical comparison are unchanged; the corrections narrow or update individual results. Where a value was refined, the original is kept in the appendix and marked as such (rows 6, 7 and 11).

#ItemSubmissionRevisionSourceWhere
1Four-band headline"lower band MAE ranks a family with 0/15 correct topologies ahead of one with 15/15 correct"; other metrics only in App. Table 7The inversion is stated as specific to band MAE; the training objective (normalized Hermitian-coefficient MSE) is named; every other metric favours the correct cohort in 15/15 pairs, now in main-text Table 1Aa8q W2ll. 23–25, 69–73, 145–155
2Benchmark failure labelsDescribed as compositeFamily-specific Chern-label disagreements; gap, link and convergence were never folded in; 69 valid mismatches, 0 invalid, 6 unresolvedRTLq Q1ll. 308–315; App. J.1, J.3
3Ethics StatementDescribed another paperDescribes this workAa8q W3, RTLq Q3ll. 548–552
4Ta₃AlC₂, six planes6/6 stable agreements, 2 nontrivial5 resolution-stable agreements, including both nontrivial planes; kz=0k_z=0 counted as unresolved (same label at both endpoints from 32×1732\times17 upward, but the label alternates with resolution)re-verificationll. 395–398; App. E.3, J.11
5Ca₃Si₄Ir₄Invalid prediction (plane gap 1.546 → 0.000617 meV)Still invalid; in addition the reference is itself nearly gapless in 3D (5 μeV on 24324^3) and its zero-planes are unresolvedre-verificationll. 359–363; App. G.2, G.5
6Two-band Fourier recoveryIID 3/15, loops 8/15 correct (96296^2)After bounded refinement: IID 2/13/0, loops 7/7/1 (correct/wrong/unresolved); original counts kept in Table 7re-verificationll. 157–161; App. D.1
7Seed-97 Fourier exampleC=−3C=-3 (96296^2)C=−4C=-4 on 1282128^2 and 1922192^2; the 96296^2 map is kept to illustrate the mechanismre-verificationFig. 2, ll. 190–192
8Rank-two witness, local maxima0.5000007–0.50000880.50000007, 0.50000014, 0.50000019; the excess over 1/2 is at single-precision scalere-verificationApp. C.2
9Rescue-cohort stability of 100%Stated for the regularized modelsHolds for the 14 regularized models that are nominally correct; the remaining one is reported separatelyre-verificationApp. C.1
10Dirichlet regularization12/15 → 14/15Unchanged; the paired changes are reported as three corrections and one regressionre-verificationApp. D.3, Table 9
11NaRuO₂, first label changeλ=0.08\lambda=0.08 eV (coarse sweep)Kept; a refined sweep brackets the change between 0.072 and 0.073 eVre-verificationll. 401–406; App. J.7, Table 29
12NaRuO₂, Frobenius-random gapsMinimum 136.7 meV130.7–153.1 meV on 64264^2 over both amplitudes; all 40 settings still keep C=−2C=-2re-verificationApp. E.2, Table 11

Rows 1, 2, 4 and 5 now state each result with the metric it concerns and its validity status; Table 20 summarizes the status of every empirical result.

AuthorsOpenReview ↗

General response (3/3): other revisions and evidence ledger

4. Other revisions

  • Introduction rewritten along one line: the problem, why it matters and is hard, what existing work leaves open, and our approach; four contributions are listed at ll. 92–107.
  • Sec. 3: the full metric comparison moved into the main text (Table 1), with the training objective named (ll. 145–155).
  • Sec. 4: the finite-range bound (ll. 212–215), the practical meaning of the 1/r1/r factor (ll. 230–234), the pointwise gap certificate (ll. 246–249), and a computed regularity bound for NaRuO₂ (ll. 267–269).
  • Sec. 5: label definitions and the validity split (ll. 308–323).
  • Sec. 6 rewritten: common-grid released audit (Table 2), Ca₃Si₄Ir₄, strained BiTeI (Table 3), Ta₃AlC₂ as a nontrivial positive control, and NaRuO₂ with 160 controls (ll. 339–416).
  • Sec. 7: the dose–response and the regions where a topology-changing error must act (ll. 419–471).
  • Sec. 9: a validity-first evaluation sequence (ll. 510–522) and a new paragraph on calibration and uniform training (ll. 525–539).
  • Fig. 4: the 0.25-meV line is labelled as the operational validity guardrail, not a phase boundary (l. 458).
  • Ethics Statement replaced (ll. 548–552).
  • Appendix J (new, 11 sections): benchmark labels; released full-HH errors; validity split; residual amplification; pointwise certificate and inversion regions; localization; NaRuO₂ refined sweep and gap enclosure; max-error training; ensembles; strained BiTeI; gap-clearance audit.
  • Supplement: round2/ holds the data and scripts for every new result; its README states, for each table and figure, whether it is recomputed or read from frozen records (ll. 557–560).

5. Evidence ledger at a glance (Table 20)

Result (Sec.)EvidenceOutcome
Band-MAE inversion (3)30 four-band models, 15 matched pairsIID 0/15, loops 15/15 correct; IID lower band MAE in 12/15 pairs
Rank-two witness (3)the 15 wrong modelsmax ff = 0.497–0.500 on 3842384^2, consistent with sup⁡f≥1/2\sup f\ge1/2
Query benchmark (5)216 instances, 7 familiesloop − stratified AUROC −0.001 (216), +0.005 (198 valid)
Released audit (6)4 released pairsZrSiPt, NaCaTaTiO₆ 6/6 agree; Ta₃AlC₂ 5 agree; Ca₃Si₄Ir₄ invalid
Strained BiTeI (6)17 structures, our DFT15 agree, 1 unresolved, 1 valid mismatch
NaRuO₂ direction (6)32-orbital Wannier modelCC changes at 72–73 meV; 160 controls keep C=−2C=-2; reference gap certified
Fixed-matrix stress (7)168 JARVIS materialsmedian band MAE 601, 556, 2262 meV at matrix MAE 1.417 meV
Band-matched stress (7)24 materials, 10–200 meVnone at 10–20 meV; GeBi₄Te₇ mismatches confirmed on dense grids
Inversion regions (7)5 dense-verified planes; Ta₃AlC₂0.02–0.45% of the plane; 4%
Max-error training, ensembles (9)four-band IID cohort0/15 correct; ensembles unanimously wrong in 15/15
Propositions 1–4 (4)proofs (App. B)proved

We thank the reviewers again; their questions led directly to the strongest new evidence in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.