- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes “zentropy” as a model-agnostic loss function in GNNs. Motivated by Helmholtz free energy, the authors replace the standard variables in the cross entropy loss with a multihead readout and additional nonlinearity. The Experiments include a wide range of graph classification tasks
Critical strengths
- The experiments are extensive and include a wide range of node- and graph-level classification datasets on a wide range of architectures
Critical weaknesses
- The proposed Zentropy should not be viewed as a modification to the cross entropy loss, but a modification to the GNN's last layer. Most loss functions do not have additional learnable parameters, which is not the case for Zentropy. Although motivated by Thermodynamics, the proposed method adds too much unjustified / weakly justified complexity that needs to be carefully disentangled and ablated. Furthermore, the additional parameters introduced can be collapsed into a (not exactly) bilinear-like MLP. Therefore the performance gain seems to mostly come from the expressivity gain of the additional bilinear-like head, which the other baseline architectures do not have, and is not particularly novel despite the physical motivations.
- The paper could be better written in a more logical manner, making sure the variables used in equations are well-introduced before or not far after the statement. Plots should be re-plotted for better clarity rather than e.g. aetheticity, with a more consistent layout. Captions and discussions should avoid extensively repeating the results in the figures.
Questions for authors
Can the authors please clarify what Figure 1a and 1b are plotting on the x axis? It is unclear how the term in causes a change in the score behaviour with increasing
4 comments
One global temperature whose heads read the same two structural descriptors matches the full model ( points, ). The temperature network is thus the route by which node structure reaches the readout.
W1(c). "The additional parameters introduced can be collapsed into a (not exactly) bilinear-like MLP. Therefore the performance gain seems to mostly come from the expressivity gain of the additional bilinear-like head, which the other baseline architectures do not have."
A: We have conducted four experiments to address your concern. Four results show that the gain does not come from head expressivity, apart from SGC and the structural inputs on the five classic heterophilic graphs at .
-
(1) Same deployed network, different training. The subgraph branch and the anchoring term exist only during training and are discarded at inference, so the models trained with and without them have identical test-time networks. Across eight settings with ten seeds each (Table R1 in our response to Reviewer sSDA), the training terms improve this identical network significantly in 6 and hurt it significantly in none. Examples are Squirrel/GCNII (27.4% → 35.9%), Cornell/GATv2 at (42.7% → 52.4%) and Pubmed/GCN at (39.4% → 73.1%). On ten further settings, the same comparison is significant in 7 and never significantly negative ( points). A difference between identical networks cannot be a difference in expressivity.
-
(2) The zentropy readout alone performs like cross-entropy. In the same factorial, the readout with all four components switched off reduces to a cross-entropy with one learned temperature and gains nothing. It reaches 26.3% against 26.6% for cross-entropy on Squirrel/GCNII, 18.6% against 18.1% on Citeseer/GATv2 (), and 40.4% against 39.6% on Pubmed/GCN (). The gain therefore comes from the components of the objective.
-
(3) Capacity-matched heads. A cross-entropy MLP head whose parameter count matches the adapter's to within 1% is significantly worse than Z-GNN in 12 of 27 settings and better in 6, four of them SGC. SGC is a linear encoder that gains from any nonlinear head, and the paper now says so. Given the temperature network's two structural descriptors at matched parameter count, the head comes close to Z-GNN on the five classic heterophilic graphs at (4 settings significantly* for Z-GNN), and it stays behind on roman-empire, amazon-ratings and Actor (27*), on the citation graphs (14*) and in seven deep settings (5*). Two pre-registered controls account for the rest. Z-GNN whose temperature network reads no structural input still beats cross-entropy ( points, ), and on the heterophilic graphs Z-GNN beats a cross-entropy with a matched, validation-chosen loss weight ( points at , at ).
-
(4) Linear probes on frozen encoders. After training, we freeze the encoder, discard its readout and fit a fresh head. In the four stacked-encoder settings whose cross-entropy collapse does not depend on dropout, Z-GNN encoders reach 37.5–69.5% with a linear probe (e.g., 61.7% on Citeseer/GATv2 at ), while cross-entropy encoders stay at 18.1–37.9% whatever head is fitted ( in each setting and readout). At the shared configuration, the improvement therefore lies in the encoder. It comes from the loss weight of the objective, and once each objective is tuned separately at depth, cross-entropy reaches the same accuracy (response to Reviewer sSDA, W1(d)).
W1(d). "[...] is not particularly novel despite the physical motivations."
A: To clarify, our novelty does not lie in the head as an architecture. In cross-entropy, a temperature rescales every class score by the same factor, as in temperature scaling for calibration (Guo et al., 2017). It can sharpen or soften a prediction but never change the predicted class (Figure 1a). Our contribution is two-fold. (1) A class-wise intrinsic entropy lets the temperature change the predicted class at the crossing temperature , and only where the entropy contrast is non-zero. (2) A training-only structural route, the subgraph branch, sets that temperature per node and improves a network that is identical at test time. The factorial quantifies both. The shared-entropy ablation confirms the first, since the class switches vanish together with the contrast, and the node-level analysis in our response to Reviewer Jog2 shows that the resulting re-rankings are mostly corrections. In the abstract and the introduction, we have also replaced the term "physics-informed neural network" with a more precise description of what the network does. Old (abstract): "we devise a physics-informed neural network to discover the critical temperature for each graph node". Revised: "we devise a physics-inspired temperature network that learns a temperature for each graph node from its representation and local structure".
As for the cost of this complexity, the adapter adds 2.6–6.8% parameters on the small graphs. On ogbn-arxiv, a 32-wide temperature network leaves accuracy unchanged ( points over 24 settings, 1 significantly lower and 0 higher), keeps Z-GNN points ahead of cross-entropy, and costs 7–9% extra training time per epoch. Since the reviewer notes that the baselines lack this head, we also tuned each objective separately on the five classic heterophilic graphs. Z-GNN remains ahead in 20 of 35 settings under an equal-budget random search and in 27 of 35 under a 27-configuration grid, with 9 significantly ahead and none significantly behind in both cases.
W2(a). "The paper could be better written in a more logical manner, making sure the variables used in equations are well-introduced before or not far after the statement."
A: Thank you for this constructive suggestion. We have revised Section 2 accordingly. (1) Every symbol is now defined before or at its first use, and a notation table is added to the appendix and referenced at the start of Section 2. (2) The effective entropy is now also defined in place in Section 2.4, next to its first appearance in the free-energy expression of Section 2.2. (3) The index typos in Eq. (1) are corrected. (4) The descriptions of the entropy-head input, , the temperature anchors and the objective are made more precise (General response).
W2(b). "Plots should be re-plotted for better clarity rather than e.g. aestheticity, with a more consistent layout."
A: Thank you for this suggestion. We have re-plotted all three figures from the runs of the revised protocol. Figure 1 is redrawn with both axes labelled, the sign of the printed formula corrected and panel (a) showing the curves (details under Q below). Figure 2 (Section 3) now uses one layout for all ten panels. Each panel is a heatmap with the seven backbones as rows in the same order, one shared colour scale for the accuracy difference, and one significance code (small and large dots for and ). The OGB and graph-classification panels, previously a scatter plot and a transposed heatmap, follow the same layout, and every cell value is listed in the appendix tables. Figure 3 (Section 4) uses the same font sizes and panel labels as Figure 2 and one colour code in all five panels, orange for the mechanism (node-wise temperature, degree partition, full model) and grey for its control. Flat markers replace the shaded bubbles, and the class-switch fraction in panel (e) is given by the printed numbers alone.
W2(c). "Captions and discussions should avoid extensively repeating the results in the figures."
A: Thank you for this suggestion. The captions now state only what each panel plots and how to read it, and each number is given once, in the text.
Q. Can the authors please clarify what Figure 1a and 1b are plotting on the x axis? It is unclear how the term in causes a change in the score behaviour with increasing .
A: We apologise for the confusion. The x-axis of both panels is the temperature . The y-axis is each label's score, i.e., its log-probability up to the shared log-partition term, which is in panel (a) and in panel (b). The formula printed in the panels lost a minus sign and should read . We have corrected it (please check Figure 1 in the updated manuscript), labelled both axes and redrawn panel (a) to show the curves explicitly, and the caption now states the axes.
In panel (a), there is no entropy term (), so every score is . As grows, all scores shrink toward 0 at the same rate, so their order never changes, and the argmax is Agents (the lowest energy) at every temperature. This is the cross-entropy case, where a temperature rescales confidence but cannot change the prediction.
In panel (b), each label adds its own , which does not depend on , while the energy part decays as . Energy therefore dominates at low and entropy at high . Two labels and swap where their scores are equal, , which gives . With the values in the figure, the swaps occur for Agents/DB at , for DB/ML at , and for ML/AI at , which are the three transitions shown. The class-wise entropy is thus what allows the temperature to change the ranking, and a node-wise temperature then selects, per node, which side of each transition it sits on.
We hope that these responses and the new experiments address the reviewer's concerns about soundness. We have uploaded a new manuscript that incorporates all the suggestions () for your review.
We thank the reviewer for the careful reading and for acknowledging the breadth of our experiments. The central concern (W1) is that the gain may come from a more expressive head, and we test this directly below.
Below, W and Q denote the reviewer's weaknesses and questions, and A denotes our answers.
A: We thank the reviewer's feedback, but we wish to clarify the core focus of our work. Our primary contribution is introducing a free-energy theoretical framework to mitigate over-smoothing in GNNs, in which a node-wise temperature helps the readout retain the node-level structure that over-smoothing tends to blur. Following this principle, we derived the zentropy loss, which elegantly encompasses standard cross-entropy as a special case when every class has zero intrinsic entropy and the temperature is one (Section 2). The learning-based approach was introduced primarily to scale optimal local temperature estimation to large graphs, and most of the gain does not depend on it. As shown in our results, with one temperature shared by all nodes and no temperature network, zentropy stays ahead of cross-entropy in 7 of the 8 factorial settings of W1(b) and level in the eighth (+10.8 points on average, against +15.0 for the full model). The temperature network, which holds most of the added parameters, contributes +1.4 points on average, against +10.4 for the subgraph branch, which adds no parameter at test time. We hope the reviewer will view our work through the lens of this theoretical development and the fundamental potential of zentropy in GNNs, beyond a simple comparison of learning and non-learning approaches.
A: Thank you for this suggestion. To disentangle the components, we replaced the sequential ablation of the submission with a full 24 factorial over the four components (class-wise entropy contrast S, node-wise temperature T, subgraph branch B, anchoring term A) on eight settings with ten seeds per arm, and we report each component's marginal contribution averaged over all orders of inclusion (Table R1 in our response to Reviewer sSDA). We can observe that (1) the subgraph branch contributes +10.4 points on average (significant in 8 of 8 settings), the entropy contrast +3.6 (6 of 8) and the node-wise temperature +1.4 (5 of 8). The node-wise temperature matters on the heterophilic settings (+2.2, significant in 5 of 5) and has little effect on the deep citation ones (+0.1). (2) The anchoring term plays a minor role. Its contribution is −0.1 points on average (significantly positive in one setting and negative in two), and removing it matters in only one of the eight settings (Squirrel/APPNP, 33.0% → 25.8%). On ten further settings, its share is never significantly positive and is slightly negative in four (−0.3 to −0.7 points). Section 4 now presents it as an optional stabiliser. (3) The output-norm penalty is not the source of the gain. Applied to cross-entropy with the same weight, it changes accuracy by −3.3 to +3.9 points, and its only significant effect is a decrease.
Each of the three remaining components has a specific and testable role. (1) Entropy contrast. Removing it alone, while keeping the parameterisation and every training term, removes the structural gain. The fraction of nodes whose class changes between the coldest and the hottest subgraph drops to zero, and the gain of a degree partition over a random one falls from +4.8 to −0.2 points on Squirrel and from +1.4 to +0.3 on roman-empire and Chameleon. This is what the theory predicts, since a temperature can re-rank classes only through a non-zero entropy contrast. (2) Subgraph branch. With K≥2 subgraphs, a degree partition beats a permuted one by +4.8 to +5.5 points on Squirrel, +1.4 to +3.6 on Cornell, +1.1 to +1.7 on roman-empire and +1.0 to +1.7 on Chameleon, and the gain stays flat over an eightfold range of K. With K=1 the two partitions coincide, and their difference (−0.03 to +0.31 on all six graphs) gives the noise floor of this measurement. A pre-registered control further shows that, on twelve heterophilic settings, the branch adds +1.70 points over a model without it whose weight decay is lowered to match the branch's loss weight. (3) Node-wise temperature. The degree partition helps significantly only together with node-wise temperatures (e.g., +4.4 on Squirrel and +2.4 on Cornell), and one global temperature removes these gains. Another pre-registered control shows what the node-wise temperature carries.