acceptodds
Under review as a conference paper at ICLR 2027

Turbocharging Graph Neural Networks with Zentropy

Abstract

Graph neural networks (GNNs) learn by aggregating feature representations across connected nodes, a mechanism that excels when graph topology aligns with label consensus. In heterophilic graphs, however, the homophily inductive bias of GNNs often yields suboptimal predictions, as the learning dynamics overemphasize relational smoothness. The popular driving force of GNN is cross-entropy, an objective that minimizes information uncertainty via a zero-temperature energy descent. Because cross-entropy does not account for internal disparities during feature aggregation, current GNN architectures remain trapped in rigid energy regimes that cannot dynamically adapt to node-level structural divergence. To address this limitation, we reformulate GNN optimization through zentropy, a multi-scale thermodynamic framework under a free-energy landscape that explicitly balances internal energy (the likelihood associated with downstream tasks) with intrinsic disparities (a prior on data heterogeneity). In the spirit of thermodynamic principle, we devise a physics-informed neural network to discover the critical temperature for each graph node, creating a new degree of freedom that mitigates representational collapse caused by feature over-smoothing in GNNs. Together, we present a model-agnostic zentropy loss function in the regime of Helmholtz free energy that serves as a plug-and-play module compatible with standard GNN architectures. We evaluate the zentropy-based readout against matched cross-entropy baselines on seven encoders, three citation networks, five heterophilic benchmarks, three OGB graphs and five graph-classification datasets. Our proposed zentropy loss rescues deep encoders in GNN where standard cross-entropy optimization collapses to chance-level performance. Across all evaluated combinations of backbone architectures and network configurations, zentropy loss yields promising performance gains, most notably on the heterophilic graph datasets.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph Neural Netwrok, Zentropy, Cross-entropy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hK66OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes “zentropy” as a model-agnostic loss function in GNNs. Motivated by Helmholtz free energy, the authors replace the standard variables in the cross entropy loss with a multihead readout and additional nonlinearity. The Experiments include a wide range of graph classification tasks

Critical strengths
  1. The experiments are extensive and include a wide range of node- and graph-level classification datasets on a wide range of architectures
Critical weaknesses
  1. The proposed Zentropy should not be viewed as a modification to the cross entropy loss, but a modification to the GNN's last layer. Most loss functions do not have additional learnable parameters, which is not the case for Zentropy. Although motivated by Thermodynamics, the proposed method adds too much unjustified / weakly justified complexity that needs to be carefully disentangled and ablated. Furthermore, the additional parameters introduced can be collapsed into a (not exactly) bilinear-like MLP. Therefore the performance gain seems to mostly come from the expressivity gain of the additional bilinear-like head, which the other baseline architectures do not have, and is not particularly novel despite the physical motivations.
  2. The paper could be better written in a more logical manner, making sure the variables used in equations are well-introduced before or not far after the statement. Plots should be re-plotted for better clarity rather than e.g. aetheticity, with a more consistent layout. Captions and discussions should avoid extensively repeating the results in the figures.
Questions for authors

Can the authors please clarify what Figure 1a and 1b are plotting on the x axis? It is unclear how the SS term in p←eE/T+Sp\leftarrow e^{E/T+S} causes a change in the score behaviour with increasing TT

4 comments

We thank the reviewer for the careful reading and for acknowledging the breadth of our experiments. The central concern (W1) is that the gain may come from a more expressive head, and we test this directly below.

Below, W and Q denote the reviewer's weaknesses and questions, and A denotes our answers.

W1(a). "The proposed Zentropy should not be viewed as a modification to the cross entropy loss, but a modification to the GNN's last layer. Most loss functions do not have additional learnable parameters."

A: We thank the reviewer's feedback, but we wish to clarify the core focus of our work. Our primary contribution is introducing a free-energy theoretical framework to mitigate over-smoothing in GNNs, in which a node-wise temperature helps the readout retain the node-level structure that over-smoothing tends to blur. Following this principle, we derived the zentropy loss, which elegantly encompasses standard cross-entropy as a special case when every class has zero intrinsic entropy and the temperature is one (Section 2). The learning-based approach was introduced primarily to scale optimal local temperature estimation to large graphs, and most of the gain does not depend on it. As shown in our results, with one temperature shared by all nodes and no temperature network, zentropy stays ahead of cross-entropy in 7 of the 8 factorial settings of W1(b) and level in the eighth (+10.8+10.8 points on average, against +15.0+15.0 for the full model). The temperature network, which holds most of the added parameters, contributes +1.4+1.4 points on average, against +10.4+10.4 for the subgraph branch, which adds no parameter at test time. We hope the reviewer will view our work through the lens of this theoretical development and the fundamental potential of zentropy in GNNs, beyond a simple comparison of learning and non-learning approaches.

W1(b). "Although motivated by Thermodynamics, the proposed method adds too much unjustified / weakly justified complexity that needs to be carefully disentangled and ablated."

A: Thank you for this suggestion. To disentangle the components, we replaced the sequential ablation of the submission with a full 242^4 factorial over the four components (class-wise entropy contrast SS, node-wise temperature TT, subgraph branch BB, anchoring term AA) on eight settings with ten seeds per arm, and we report each component's marginal contribution averaged over all orders of inclusion (Table R1 in our response to Reviewer sSDA). We can observe that (1) the subgraph branch contributes +10.4+10.4 points on average (significant in 8 of 8 settings), the entropy contrast +3.6+3.6 (6 of 8) and the node-wise temperature +1.4+1.4 (5 of 8). The node-wise temperature matters on the heterophilic settings (+2.2+2.2, significant in 5 of 5) and has little effect on the deep citation ones (+0.1+0.1). (2) The anchoring term plays a minor role. Its contribution is −0.1-0.1 points on average (significantly positive in one setting and negative in two), and removing it matters in only one of the eight settings (Squirrel/APPNP, 33.0% → 25.8%). On ten further settings, its share is never significantly positive and is slightly negative in four (−0.3-0.3 to −0.7-0.7 points). Section 4 now presents it as an optional stabiliser. (3) The output-norm penalty is not the source of the gain. Applied to cross-entropy with the same weight, it changes accuracy by −3.3-3.3 to +3.9+3.9 points, and its only significant effect is a decrease.

Each of the three remaining components has a specific and testable role. (1) Entropy contrast. Removing it alone, while keeping the parameterisation and every training term, removes the structural gain. The fraction of nodes whose class changes between the coldest and the hottest subgraph drops to zero, and the gain of a degree partition over a random one falls from +4.8+4.8 to −0.2-0.2 points on Squirrel and from +1.4+1.4 to +0.3+0.3 on roman-empire and Chameleon. This is what the theory predicts, since a temperature can re-rank classes only through a non-zero entropy contrast. (2) Subgraph branch. With K≥2K \ge 2 subgraphs, a degree partition beats a permuted one by +4.8+4.8 to +5.5+5.5 points on Squirrel, +1.4+1.4 to +3.6+3.6 on Cornell, +1.1+1.1 to +1.7+1.7 on roman-empire and +1.0+1.0 to +1.7+1.7 on Chameleon, and the gain stays flat over an eightfold range of KK. With K=1K=1 the two partitions coincide, and their difference (−0.03-0.03 to +0.31+0.31 on all six graphs) gives the noise floor of this measurement. A pre-registered control further shows that, on twelve heterophilic settings, the branch adds +1.70+1.70 points over a model without it whose weight decay is lowered to match the branch's loss weight. (3) Node-wise temperature. The degree partition helps significantly only together with node-wise temperatures (e.g., +4.4+4.4 on Squirrel and +2.4+2.4 on Cornell), and one global temperature removes these gains. Another pre-registered control shows what the node-wise temperature carries.

One global temperature whose heads read the same two structural descriptors matches the full model (+0.17+0.17 points, [−0.27,+0.57][-0.27, +0.57]). The temperature network is thus the route by which node structure reaches the readout.

W1(c). "The additional parameters introduced can be collapsed into a (not exactly) bilinear-like MLP. Therefore the performance gain seems to mostly come from the expressivity gain of the additional bilinear-like head, which the other baseline architectures do not have."

A: We have conducted four experiments to address your concern. Four results show that the gain does not come from head expressivity, apart from SGC and the structural inputs on the five classic heterophilic graphs at L=2L=2.

  • (1) Same deployed network, different training. The subgraph branch and the anchoring term exist only during training and are discarded at inference, so the models trained with and without them have identical test-time networks. Across eight settings with ten seeds each (Table R1 in our response to Reviewer sSDA), the training terms improve this identical network significantly in 6 and hurt it significantly in none. Examples are Squirrel/GCNII (27.4% → 35.9%), Cornell/GATv2 at L=16L=16 (42.7% → 52.4%) and Pubmed/GCN at L=8L=8 (39.4% → 73.1%). On ten further settings, the same comparison is significant in 7 and never significantly negative (+6.0+6.0 points). A difference between identical networks cannot be a difference in expressivity.

  • (2) The zentropy readout alone performs like cross-entropy. In the same factorial, the readout with all four components switched off reduces to a cross-entropy with one learned temperature and gains nothing. It reaches 26.3% against 26.6% for cross-entropy on Squirrel/GCNII, 18.6% against 18.1% on Citeseer/GATv2 (L=4L=4), and 40.4% against 39.6% on Pubmed/GCN (L=8L=8). The gain therefore comes from the components of the objective.

  • (3) Capacity-matched heads. A cross-entropy MLP head whose parameter count matches the adapter's to within 1% is significantly worse than Z-GNN in 12 of 27 settings and better in 6, four of them SGC. SGC is a linear encoder that gains from any nonlinear head, and the paper now says so. Given the temperature network's two structural descriptors at matched parameter count, the head comes close to Z-GNN on the five classic heterophilic graphs at L=2L=2 (4 settings significantly* for Z-GNN), and it stays behind on roman-empire, amazon-ratings and Actor (27*), on the citation graphs (14*) and in seven deep settings (5*). Two pre-registered controls account for the rest. Z-GNN whose temperature network reads no structural input still beats cross-entropy (+0.84+0.84 points, [+0.11,+1.65][+0.11, +1.65]), and on the heterophilic graphs Z-GNN beats a cross-entropy with a matched, validation-chosen loss weight (+1.28+1.28 points at L=2L=2, +2.44+2.44 at L=8L=8).

  • (4) Linear probes on frozen encoders. After training, we freeze the encoder, discard its readout and fit a fresh head. In the four stacked-encoder settings whose cross-entropy collapse does not depend on dropout, Z-GNN encoders reach 37.5–69.5% with a linear probe (e.g., 61.7% on Citeseer/GATv2 at L=4L=4), while cross-entropy encoders stay at 18.1–37.9% whatever head is fitted (p≤0.016p \le 0.016 in each setting and readout). At the shared configuration, the improvement therefore lies in the encoder. It comes from the loss weight of the objective, and once each objective is tuned separately at depth, cross-entropy reaches the same accuracy (response to Reviewer sSDA, W1(d)).

W1(d). "[...] is not particularly novel despite the physical motivations."

A: To clarify, our novelty does not lie in the head as an architecture. In cross-entropy, a temperature rescales every class score by the same factor, as in temperature scaling for calibration (Guo et al., 2017). It can sharpen or soften a prediction but never change the predicted class (Figure 1a). Our contribution is two-fold. (1) A class-wise intrinsic entropy lets the temperature change the predicted class at the crossing temperature T∗=ΔE/ΔST^* = \Delta E/\Delta\mathbb S, and only where the entropy contrast is non-zero. (2) A training-only structural route, the subgraph branch, sets that temperature per node and improves a network that is identical at test time. The factorial quantifies both. The shared-entropy ablation confirms the first, since the class switches vanish together with the contrast, and the node-level analysis in our response to Reviewer Jog2 shows that the resulting re-rankings are mostly corrections. In the abstract and the introduction, we have also replaced the term "physics-informed neural network" with a more precise description of what the network does. Old (abstract): "we devise a physics-informed neural network to discover the critical temperature for each graph node". Revised: "we devise a physics-inspired temperature network that learns a temperature for each graph node from its representation and local structure".

As for the cost of this complexity, the adapter adds 2.6–6.8% parameters on the small graphs. On ogbn-arxiv, a 32-wide temperature network leaves accuracy unchanged (−0.04-0.04 points over 24 settings, 1 significantly lower and 0 higher), keeps Z-GNN +0.44+0.44 points ahead of cross-entropy, and costs 7–9% extra training time per epoch. Since the reviewer notes that the baselines lack this head, we also tuned each objective separately on the five classic heterophilic graphs. Z-GNN remains ahead in 20 of 35 settings under an equal-budget random search and in 27 of 35 under a 27-configuration grid, with 9 significantly ahead and none significantly behind in both cases.

W2(a). "The paper could be better written in a more logical manner, making sure the variables used in equations are well-introduced before or not far after the statement."

A: Thank you for this constructive suggestion. We have revised Section 2 accordingly. (1) Every symbol is now defined before or at its first use, and a notation table is added to the appendix and referenced at the start of Section 2. (2) The effective entropy S(c)=S(c)−(S(c))2/γ2\mathbb S^{(c)} = S^{(c)} - (S^{(c)})^2/\gamma^2 is now also defined in place in Section 2.4, next to its first appearance in the free-energy expression of Section 2.2. (3) The index typos in Eq. (1) are corrected. (4) The descriptions of the entropy-head input, δv\delta_v, the temperature anchors and the objective are made more precise (General response).

W2(b). "Plots should be re-plotted for better clarity rather than e.g. aestheticity, with a more consistent layout."

A: Thank you for this suggestion. We have re-plotted all three figures from the runs of the revised protocol. Figure 1 is redrawn with both axes labelled, the sign of the printed formula corrected and panel (a) showing the −Ec/T-E_c/T curves (details under Q below). Figure 2 (Section 3) now uses one layout for all ten panels. Each panel is a heatmap with the seven backbones as rows in the same order, one shared colour scale for the accuracy difference, and one significance code (small and large dots for p≤0.05p \le 0.05 and p≤0.01p \le 0.01). The OGB and graph-classification panels, previously a scatter plot and a transposed heatmap, follow the same layout, and every cell value is listed in the appendix tables. Figure 3 (Section 4) uses the same font sizes and panel labels as Figure 2 and one colour code in all five panels, orange for the mechanism (node-wise temperature, degree partition, full model) and grey for its control. Flat markers replace the shaded bubbles, and the class-switch fraction in panel (e) is given by the printed numbers alone.

W2(c). "Captions and discussions should avoid extensively repeating the results in the figures."

A: Thank you for this suggestion. The captions now state only what each panel plots and how to read it, and each number is given once, in the text.

Q. Can the authors please clarify what Figure 1a and 1b are plotting on the x axis? It is unclear how the SS term in p←eE/T+Sp \leftarrow e^{E/T+S} causes a change in the score behaviour with increasing TT.

A: We apologise for the confusion. The x-axis of both panels is the temperature TT. The y-axis is each label's score, i.e., its log-probability up to the shared log-partition term, which is −Ec/T-E_c/T in panel (a) and −Ec/T+Sc-E_c/T + S_c in panel (b). The formula printed in the panels lost a minus sign and should read p∝e−E/T+Sp \propto e^{-E/T + S}. We have corrected it (please check Figure 1 in the updated manuscript), labelled both axes and redrawn panel (a) to show the −Ec/T-E_c/T curves explicitly, and the caption now states the axes.

In panel (a), there is no entropy term (S=0S = 0), so every score is −Ec/T-E_c/T. As TT grows, all scores shrink toward 0 at the same rate, so their order never changes, and the argmax is Agents (the lowest energy) at every temperature. This is the cross-entropy case, where a temperature rescales confidence but cannot change the prediction.

In panel (b), each label adds its own ScS_c, which does not depend on TT, while the energy part decays as 1/T1/T. Energy therefore dominates at low TT and entropy at high TT. Two labels cc and dd swap where their scores are equal, −Ec/T+Sc=−Ed/T+Sd-E_c/T + S_c = -E_d/T + S_d, which gives T∗=ΔE/ΔST^* = \Delta E/\Delta S. With the values in the figure, the swaps occur for Agents/DB at 0.15/0.30=0.50.15/0.30 = 0.5, for DB/ML at 1.00/1.00=1.01.00/1.00 = 1.0, and for ML/AI at 3.00/1.50=2.03.00/1.50 = 2.0, which are the three transitions shown. The class-wise entropy is thus what allows the temperature to change the ranking, and a node-wise temperature then selects, per node, which side of each transition it sits on.

We hope that these responses and the new experiments address the reviewer's concerns about soundness. We have uploaded a new manuscript that incorporates all the suggestions (highlighted in blue\textcolor{blue}{\text{highlighted in blue}}) for your review.

Reviewer_Jog2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper incorporate zentropy into GNN objective combining learned class energies, intrinsic entropies, node specific temperatures and degree-based subgraph regularisation. The extensive experiments show substantial improvements in some heterophilic and deep GNN settings. However, there is deviation from the true goal of superiority in terms of the expressiveness of this new objective compared to cross entropy based GNNs.

Critical strengths

The proposed objective works with multiple GNN architectures. The depth evaluations, component ablations in Fig. 3 and frozen encoder experiments in Table 7 provide useful information to support the objective can improve learned representations. Theoretic analysis is provided to support the argument.

Critical weaknesses

Practioners are interested in whether the correct class label can be made with max probability instead of swapping non-critical labels. Yes the analysis shows sawpping is possible when the temperature value changes. However, is the ultimate goal achieved and break free from the homophily? The paper somehow deviated from this although the experimental results seem to support it empirically.

Questions for authors

It would be better to address the concern raised in weakness section.

3 comments

Thanks for the responses. My score remains the same.

Dear Reviewer Jog2,

Thank you very much for taking the time to read our rebuttal and engage in the discussion.

We noticed that your score remains unchanged, and we would greatly appreciate it if you could let us know whether our additional experiments (such as the node-level re-ranking analysis, the AMI predictor, and the new heterophilic benchmarks) and revisions have addressed your original concerns. If there are any remaining questions or points that you feel are still unclear, we would be more than happy to clarify them further before the discussion period ends.

If our response has resolved your concerns, we would kindly ask if you might reconsider updating your rating to reflect the improvements made in the revised manuscript.

Thank you again for your time and valuable feedback.

Best regards,

Authors of Submission 47671

AuthorsOpenReview ↗

Paper title: Turbocharge graph neural networks with Zentropy

Dear Reviewer Jog2, As the discussion period is coming to a close, we would like to kindly follow up on our responses, in which we addressed each of your comments with additional experiments. We would greatly appreciate your feedback. Thank you again for your time and valuable comments.

Best regards, The Authors

Reviewer_sSDAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a zentropy-inspired GNN readout with class-dependent energy and entropy heads and a learned node-specific temperature. The entropy and temperature heads use representations augmented by degree and neighborhood disagreement. Training combines node-temperature and degree-partition-temperature losses with regularization. Evaluation spans several backbones and depths, citation and heterophilic benchmarks, OGB graphs, and graph classification. Ablations examine partitioning, entropy contrast, the composite objective, and frozen-encoder probes.

Critical strengths
  1. The structural conditioning is plausible and evaluated across several settings. Degree and neighborhood disagreement can provide context beyond a standard linear head. The paper reports improvements at shallow as well as deep settings and includes negative and small effects. The contribution is therefore worth considering beyond the largest examples of recovery from degraded deep baselines.

  2. The ablations provide useful evidence about parts of the mechanism. Table 7's frozen-encoder experiments show that some improvements remain with a new linear probe, supporting an effect on the encoder during training beyond final-head capacity. Figure 3(e) reports that temperature-induced class switching disappears with a shared entropy head, as expected when classwise entropy contrast vanishes. These are relevant checks, although they do not fully attribute the predictive gains to individual training components.

Critical weaknesses
  1. The source of the improvement remains insufficiently separated from the additional training components and baseline configuration. For GCNII on Squirrel, Figure 3(b) reports accuracy increasing from 26.1% with cross-entropy to 27.3% with the zentropy readout and node temperatures, then to 34.4% with the full objective. These sequential results suggest that the additional components help in that experiment; they do not assign independent causal shares to interacting components. A cross-entropy control with the same output-norm regularization would help test a relevant alternative explanation. The shared hyperparameters in Section 3.1 support a matched-configuration comparison, but do not establish gains over separately tuned alternatives. Table 3 also reports CiteSeer APPNP decreasing from 68.28% at depth 2 to 22.68% at depth 4. Clarifying what depth changes and specifying the restart and encoder settings would help interpret this recovery example. These observations alone establish neither a baseline implementation error nor the cause of its degradation.

  2. The theoretical explanation needs reconciliation with the actual readout. Subtracting the class log-probabilities in Eq. 2 gives the following algebraic expression, with k_B = 1:

log⁡(pc/pd)=−(Ec−Ed)/T+(Sc−Sd)−(Sc2−Sd2)/γ2\log(p_c/p_d) = -(E_c - E_d)/T + (S_c - S_d) - (S_c^2 - S_d^2)/\gamma^2

Section 2.4 omits the last term, so the stated crossing thresholds need an effective-entropy definition or a corrected expression when its coefficient is nonzero. Appendix Table 1 lists the coefficient as 0.00 on Cora and 0.01 on CiteSeer and Pubmed. The term vanishes when the coefficient is exactly zero; otherwise its size depends on the learned entropies. The result that identical class entropies prevent temperature-induced re-ranking still holds. Separately, Proposition 1's bound is valid for a readout Lipschitz in h, but the proposed head also receives degree and neighborhood disagreement. Identical h values can coexist with different degrees and outputs. Its application to the augmented head therefore needs an additional argument or a narrower collapse claim.

Questions for authors
  1. In a representative improvement setting, how does cross-entropy perform with the same output-norm regularization? For the CiteSeer APPNP example, please specify the restart coefficient and whether depth changes propagation steps, encoder layers, or both, and clarify whether the gain persists under an appropriate configuration sensitivity check.
  2. Can you reconcile Section 2.4's entropy and threshold definitions with Eq. 2, and explain how Proposition 1 applies when the readout receives structural descriptors in addition to h? Please distinguish changes to the explanatory claims from any changes needed in the implementation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.