- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper analyzes how different representations of textual node features effect performance of GNNs on heterogeneous text-attributed graphs (HTAGs). The paper introduces a benchmark for this consisting of 6 HTAGs (all sourced from prior works) and 15 text encodings, from a random baseline and simple TF-IDF to LLM embeddings. The paper analyzes how changing textual representations affects the performance of both homogeneous GNNs and heterogeneous GNNs and provides a number of interesting observations. The paper also introduces Text-Graph Alignment (TGA) score - a training-free label-free measure that correlates well with the gains obtained by homogeneous GNNs.
Critical strengths
The paper is very well-written and easy to follow.
The conclusions obtained from the analysis a quite interesting and novel, and thus might be useful for the field assuming the analysis is correct.
Critical weaknesses
I believe there is a critical flaw in the methodology used for the analysis which makes the obtained insights unreliable. Much of the analysis relies on comparing gains obtained by GNNs over a simple MLP with different textual representations. However, I believe these gains are not actually comparable since obtaining an improvement of x percentage points can be more or less difficult depending on the baseline, i.e. obtaining an improvement of 10 from a baseline of 50 is most likely a lot easier than obtaining the same improvement of 10 from a baseline of 70 (because more complex instances remain after the increase in baseline performance). For example, the paper claims: "stronger representations generally reduce homogeneous headroom", but this is naturally expected, as stronger representations lead to better baseline MLP performance, and thus obtaining specific gains over this baseline becomes more difficult.
Questions for authors
Can the authors adress the issue raised in Weaknesses?
4 comments
Three Further Checks and a Note on Prior Work
Continuing from our response "A Fixed-Baseline Test of the Comparability Concern", we add three further checks and a citation we should have included.
- Same features, different graph. Degree-preserving rewiring of citation edges leaves every node's features, and hence the MLP, unchanged. The SAGEskip gain still falls by 2.00 points on DBLP and 1.38 points on arXiv (Appendix K.2). If the baseline alone set the gain, it could not change here.
- Same graph, matched baselines. Within a graph, representations whose MLP Micro-F1 differs by at most one point still differ in gain by up to 2.7 points (Appendix K.1).
- A reversal the ceiling cannot produce. A stronger baseline can shrink a gain but cannot make it negative, since a graph that adds any evidence still adds some accuracy. Yet on CroVal, SAGE adds 4.87 points over the MLP with DistilBERT features and loses 2.52 points with NV-Embed-v2 (Appendix K.1), and it falls below the MLP in all five seeds (89.32 ± 0.09 vs. 91.84 ± 0.07). With RSAGE's self skip, the same model matches the MLP (91.91). The drop therefore goes beyond a harder baseline. It can be read as the graph adding no evidence that the text lacks, with plain SAGE paying for neighbors it cannot ignore.
- Prior work. The within-class feature shuffle in point 1 of our response builds on Lee et al. [a], who showed on homogeneous graphs that it raises GNN accuracy while the MLP is unchanged. We apply it across text encoders and HTAGs, and the revision now cites it (Appendix K.6).
In short, the baseline explains part of the raw-point decline, and the paper now says so, but we confirmed that the gain still moves in every test where the baseline is held fixed. We hope these experiments and analyses resolve your concern.
These results also have implications for how HTAG benchmarks evaluate graph models. Recent HTAG benchmarks (Liu et al., 2025; Zhou et al., 2026; Zhang et al., 2026) mainly compare models and model families, rather than how the gain of a fixed graph model changes with the text representation. We therefore think evaluation should report graph-model gains together with the text-only baseline and a baseline-adjusted measure, and should include a sweep over text representations.
Your comment helped us make the logic of our claims more rigorous, and we are grateful for it. We would be glad to answer any further questions. Thank you.
References
[a] Lee et al. "Feature Distribution on Graph Topology Mediates the Effect of Graph Convolution." ICML, 2024.
I appreciate the authors' rebuttal. However, I believe the uncovered flaw in the analysis methodology requires a major paper rewrite beyond what has already been done and beyond what is likely possible during a rebuttal. Thus, I maintain my score.
Same Results and Stronger Evidence After Controlling for the Baseline
Thank you for your response, and we understand your point. We agree that the original paper did not sufficiently separate the observed SAGE–MLP gain from baseline-dependent ceiling effects. This required us to refine how the gain is interpreted, but it did not change the benchmark, the measurements, or the experimental design, and the measured results in Figure 3 and Tables 5–10 remain the same. Sections 5.2, 7.2, and 8 now clarify what those results support, and the new controls test whether the observed patterns persist when the baseline is held fixed.
The original results remain valid, and the added experiments strengthen their logic
The definition and the measurements are the same as in the original submission. The homogeneous gain is still SAGE minus MLP on the same representation (Section 5.2). In response to your concern, the revision states what this quantity does not mean (Section 5.2), separates the two causes of the contraction (Section 7.2, "Why the gain contracts"), rewords the sentence you quoted (Section 8), and states the scope of TGA more precisely, as a label-free diagnostic within a graph, not as a better ranker than a fully labeled MLP. If the contraction were only a baseline effect, it would disappear once the baseline is controlled. As shown below, it does not.
The original results already show that the baseline does not explain the contraction alone. All numbers below come from the original runs (Tables 5–10 and their saved checkpoints), without retraining.
- A baseline ceiling alone cannot explain negative gains. On CroVal, SAGE is below the MLP for 9 of the 14 text representations (Table 6), in all five seeds for each, and SAGE's best seed is below the MLP's worst.
- The baseline-adjusted gain also falls. The share of remaining errors that SAGE corrects, (SAGE − MLP)/(100 − MLP), falls as MLP accuracy rises on all six HTAGs (Spearman −0.67 to −0.99).
- A higher baseline can come with a larger gain. On TMDB, Sentence-T5 has a higher MLP than BGE-large (80.53 vs. 79.89) and a larger gain (+4.93 vs. +3.34). On CroVal, BGE-M3 has a higher MLP than TF–IDF (86.02 vs. 85.45) and a larger gain (+1.23 vs. +0.35). A ceiling alone predicts the opposite.
- Matched baselines, different gains. Within a graph, representations whose MLP accuracy differs by at most one point differ in gain by up to 2.7 points (Appendix K.1).
- The remaining errors are not harder for the graph model. On test nodes that both MLPs misclassify, strong-text SAGE is at least as accurate on all six HTAGs, using the original checkpoints (Figure 4a). Of the nodes that only the strong-text MLP solves, 49–73% were already solved by weak-text SAGE through the graph, so strong text now solves directly what the graph used to add.
- Representation selection is not affected. These results compare absolute accuracies, not gains. The default features lose 2.50 (SAGE) and 2.64 (RSAGE) points to the best of 14 representations.
What the added controls show
With the MLP identical by construction, permuting features within (label, split) groups raised the gain in all 50 settings, and across six levels of this permutation TGA computed before any GNN run ordered the gain on all five tested HTAGs (median Spearman −0.86 to −1.00). With features fixed, rewiring citations changed the gain (Appendices K.2, K.6). These controls also show that TGA carries information beyond text-only accuracy, which the original correlations alone could not establish.
We designed the controls above around the concern that raw gains over different baselines are not directly comparable. Since these controls do not overturn the original results, we believe the original measurements and experimental setup are sound. The core question and the experimental results remain, while the revision refines how the gain is interpreted and the scope of our claims.
These results are already part of the revised paper. Section 7.2 summarizes them and points to Appendix K, so the revision as it stands reports the evidence that separates our findings from the baseline effect.
Taken together, these analyses do not make raw gains directly comparable across different baselines. They show that the baseline level alone does not explain the observed variation: gains also change when the MLP is held fixed, and the baseline-adjusted and same-error analyses support the same interpretation. We have revised the paper to state this distinction and to limit our claims accordingly, and in doing so we confirmed that the problem the paper addresses and its main findings did not change. In your review, you noted that the conclusions might be useful for the field if the analysis is correct, and we hope these analyses address that condition. If you have any further questions, please let us know. Thank you again for engaging with our response, and we would be grateful if you would reconsider your assessment!
A Fixed-Baseline Test of the Comparability Concern
Thank you for raising this concern. It led us to change how the paper states and tests its main result. New results are in Appendix K of the revision (changes in blue).
The concern: gains over different baselines are not comparable
We agree with the premise. A 10-point gain from a 50-point baseline and from a 70-point baseline are not equally hard, and a stronger baseline leaves fewer and different errors. The fall in raw points follows this ceiling, and the revision says so (Sections 5.2 and 7.2). The sentence you quote now reads "reduce the observed added Micro-F1 of homogeneous message passing", and Section 5.2 states that headroom means only this observed difference. The revision no longer rests on comparing raw gains. It holds the baseline fixed, measures how much of the decline the baseline explains, and checks the same errors directly.
1. The baseline fixed by construction (Appendix K.6)
We moved target features only within (label, split) groups. Every split keeps the same (feature, label) pairs, so the MLP's accuracy is identical. Only the feature similarity between connected nodes, beyond what their labels explain, changes.
A difference in baseline cannot explain these changes, because there is none.
2. The baseline-adjusted decline and its cause
The share of remaining errors SAGEskip corrects, (Gain)/(100 − MLP), falls from the weakest to the strongest representation. After the permutation, it is nearly flat:
For multi-label Book this is a share of the remaining Micro-F1. With RSAGE's self skip added to SAGE, 91% of the raw-point decline remains (8.00 → 7.29).
How much is the baseline. We converted accuracy to evidence units with a probit link. A graph that adds constant evidence would still show a smaller raw gain over a stronger MLP. This ceiling explains 22–39% of the observed decline on CroVal, arXiv, DBLP, and Patent (37–59% under a binary link). The rest is evidence the graph no longer adds. On TMDB, where neighbor features are only weakly similar, the ceiling explains all of it. At matched MLP accuracy (70–76, 24 cells), the SAGE gain still ranges from +2.6 to +9.9 across graphs. In the model of Appendix K.6, a stronger encoder removes graph evidence only when a node's features resemble its neighbors' beyond their labels. The permutation removes this resemblance at a fixed encoder, and the gain returns.
3. The same errors, with stronger text (Figure 4a, Appendix K.1)
For the weakest and strongest representation, we took test nodes that both MLPs misclassify, so both score 0 there.
Strong-text SAGE is as accurate or more on all six datasets, and more accurate in every one of five seeds on TMDB, CroVal, and arXiv. Of the nodes only the strong MLP solves, 49–73% were already solved by weak-text SAGE, against 17–35% of the nodes both miss. Strong-text SAGE still corrects 25–56% of the strong MLP's errors. This agrees with your error-composition account and shows where it acts.
4. What the paper now claims
The fixed-baseline test addresses the concern directly: with the MLP held identical, the gain still moves by +2.5 to +12.0 points, so the decline is not an artifact of different baselines alone. We add two further checks below.