acceptodds
Under review as a conference paper at ICLR 2027

When Do Heterogeneous Text-Attributed Graphs Need Graphs?

Abstract

Heterogeneous text-attributed graphs (HTAGs) combine rich node text with typed graph structure. However, existing HTAG evaluations typically fix a single text encoder, leaving unclear how representation changes affect different structural gains. We introduce MeHTAGs Bench, which pairs six HTAG datasets with 15 node representations, including a random control, yielding 90 settings with fixed graphs, labels, target nodes, and splits. Across representation and architecture sweeps, homogeneous message-passing gains generally contract as text representations strengthen, whereas additional relation-aware gains show no consistent contraction. Moreover, the best text-only representation need not be the best graph-model input. We further introduce Text–Graph Alignment (TGA), a label-free, training-free rank statistic that compares the cosine similarities of meta-path-connected target-node pairs with those of randomly sampled pairs. TGA exhibits the strongest absolute rank correlation with residual homogeneous message-passing gains among the evaluated label-free statistics in each of the six HTAGs. Across all six HTAGs, fixed-weight aggregation probes further associate higher input TGA with smaller aggregation-induced alignment gains, even after ceiling normalization. These results show that graph utility depends on the graph–representation pair, with TGA providing an empirical diagnostic of residual homogeneous gains within a fixed HTAG.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
heterogeneous text-attributed graphs, text-attributed graphs, heterogeneous graph neural networks, text embeddings, benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_A7XSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper analyzes how different representations of textual node features effect performance of GNNs on heterogeneous text-attributed graphs (HTAGs). The paper introduces a benchmark for this consisting of 6 HTAGs (all sourced from prior works) and 15 text encodings, from a random baseline and simple TF-IDF to LLM embeddings. The paper analyzes how changing textual representations affects the performance of both homogeneous GNNs and heterogeneous GNNs and provides a number of interesting observations. The paper also introduces Text-Graph Alignment (TGA) score - a training-free label-free measure that correlates well with the gains obtained by homogeneous GNNs.

Critical strengths

The paper is very well-written and easy to follow.

The conclusions obtained from the analysis a quite interesting and novel, and thus might be useful for the field assuming the analysis is correct.

Critical weaknesses

I believe there is a critical flaw in the methodology used for the analysis which makes the obtained insights unreliable. Much of the analysis relies on comparing gains obtained by GNNs over a simple MLP with different textual representations. However, I believe these gains are not actually comparable since obtaining an improvement of x percentage points can be more or less difficult depending on the baseline, i.e. obtaining an improvement of 10 from a baseline of 50 is most likely a lot easier than obtaining the same improvement of 10 from a baseline of 70 (because more complex instances remain after the increase in baseline performance). For example, the paper claims: "stronger representations generally reduce homogeneous headroom", but this is naturally expected, as stronger representations lead to better baseline MLP performance, and thus obtaining specific gains over this baseline becomes more difficult.

Questions for authors

Can the authors adress the issue raised in Weaknesses?

4 comments
AuthorsOpenReview ↗

A Fixed-Baseline Test of the Comparability Concern

Thank you for raising this concern. It led us to change how the paper states and tests its main result. New results are in Appendix K of the revision (changes in blue).

The concern: gains over different baselines are not comparable

We agree with the premise. A 10-point gain from a 50-point baseline and from a 70-point baseline are not equally hard, and a stronger baseline leaves fewer and different errors. The fall in raw points follows this ceiling, and the revision says so (Sections 5.2 and 7.2). The sentence you quote now reads "reduce the observed added Micro-F1 of homogeneous message passing", and Section 5.2 states that headroom means only this observed difference. The revision no longer rests on comparing raw gains. It holds the baseline fixed, measures how much of the decline the baseline explains, and checks the same errors directly.

1. The baseline fixed by construction (Appendix K.6)

We moved target features only within (label, split) groups. Every split keeps the same (feature, label) pairs, so the MLP's accuracy is identical. Only the feature similarity between connected nodes, beyond what their labels explain, changes.

  • Full permutation. The gain of SAGEskip (SAGE with RSAGE's self skip) rose in all 50 dataset–representation settings on five HTAGs, by +2.5 to +12.0 points on average.
  • Six levels. TGA, computed before any GNN run, ordered the gain on all five HTAGs (median Spearman −0.86 to −1.00). Adding TGA to MLP accuracy raised R² from 0.22–0.71 to 0.86–0.93. Permuting half of the nodes in each group raised the gain on all five HTAGs (+1.6 to +5.6 points). Where our procedure raised this similarity (TMDB, Book), the gain fell (−1.26, −0.41 points).

A difference in baseline cannot explain these changes, because there is none.

2. The baseline-adjusted decline and its cause

The share of remaining errors SAGEskip corrects, (Gain)/(100 − MLP), falls from the weakest to the strongest representation. After the permutation, it is nearly flat:

DatasetOriginalPermuted
TMDB27.9% → 21.5%33.8% → 32.2%
CroVal19.9% → 0.9%39.3% → 50.5%
arXiv33.6% → 11.8%57.9% → 60.0%
Book27.1% → 10.1%44.3% → 43.8%
DBLP28.3% → 9.6%58.5% → 57.9%

For multi-label Book this is a share of the remaining Micro-F1. With RSAGE's self skip added to SAGE, 91% of the raw-point decline remains (8.00 → 7.29).

How much is the baseline. We converted accuracy to evidence units with a probit link. A graph that adds constant evidence would still show a smaller raw gain over a stronger MLP. This ceiling explains 22–39% of the observed decline on CroVal, arXiv, DBLP, and Patent (37–59% under a binary link). The rest is evidence the graph no longer adds. On TMDB, where neighbor features are only weakly similar, the ceiling explains all of it. At matched MLP accuracy (70–76, 24 cells), the SAGE gain still ranges from +2.6 to +9.9 across graphs. In the model of Appendix K.6, a stronger encoder removes graph evidence only when a node's features resemble its neighbors' beyond their labels. The permutation removes this resemblance at a fixed encoder, and the gain returns.

3. The same errors, with stronger text (Figure 4a, Appendix K.1)

For the weakest and strongest representation, we took test nodes that both MLPs misclassify, so both score 0 there.

DatasetShare of test nodesSAGE accuracy, weak → strong
TMDB8.9%34.7 → 53.1
CroVal6.0%23.7 → 33.4
arXiv13.9%20.1 → 26.2
Book (exact match)42.4%16.9 → 22.4
DBLP22.1%19.9 → 20.5
Patent20.1%30.4 → 34.0

Strong-text SAGE is as accurate or more on all six datasets, and more accurate in every one of five seeds on TMDB, CroVal, and arXiv. Of the nodes only the strong MLP solves, 49–73% were already solved by weak-text SAGE, against 17–35% of the nodes both miss. Strong-text SAGE still corrects 25–56% of the strong MLP's errors. This agrees with your error-composition account and shows where it acts.

4. What the paper now claims

  • The raw-point decline partly reflects the stronger baseline. We do not claim the direction is surprising.
  • The graph adds less when a node's own features already resemble those of its neighbors. This holds with the baseline fixed and on a baseline-adjusted measure.
  • TGA captures this feature similarity without labels. It orders the gain at fixed MLP accuracy, so it is not only a proxy for text-only accuracy.
  • We do not claim that TGA ranks gains better than a fully labeled MLP across the evaluated encoders.

The fixed-baseline test addresses the concern directly: with the MLP held identical, the gain still moves by +2.5 to +12.0 points, so the decline is not an artifact of different baselines alone. We add two further checks below.

AuthorsOpenReview ↗

Three Further Checks and a Note on Prior Work

Continuing from our response "A Fixed-Baseline Test of the Comparability Concern", we add three further checks and a citation we should have included.

  • Same features, different graph. Degree-preserving rewiring of citation edges leaves every node's features, and hence the MLP, unchanged. The SAGEskip gain still falls by 2.00 points on DBLP and 1.38 points on arXiv (Appendix K.2). If the baseline alone set the gain, it could not change here.
  • Same graph, matched baselines. Within a graph, representations whose MLP Micro-F1 differs by at most one point still differ in gain by up to 2.7 points (Appendix K.1).
  • A reversal the ceiling cannot produce. A stronger baseline can shrink a gain but cannot make it negative, since a graph that adds any evidence still adds some accuracy. Yet on CroVal, SAGE adds 4.87 points over the MLP with DistilBERT features and loses 2.52 points with NV-Embed-v2 (Appendix K.1), and it falls below the MLP in all five seeds (89.32 ± 0.09 vs. 91.84 ± 0.07). With RSAGE's self skip, the same model matches the MLP (91.91). The drop therefore goes beyond a harder baseline. It can be read as the graph adding no evidence that the text lacks, with plain SAGE paying for neighbors it cannot ignore.
  • Prior work. The within-class feature shuffle in point 1 of our response builds on Lee et al. [a], who showed on homogeneous graphs that it raises GNN accuracy while the MLP is unchanged. We apply it across text encoders and HTAGs, and the revision now cites it (Appendix K.6).

In short, the baseline explains part of the raw-point decline, and the paper now says so, but we confirmed that the gain still moves in every test where the baseline is held fixed. We hope these experiments and analyses resolve your concern.

These results also have implications for how HTAG benchmarks evaluate graph models. Recent HTAG benchmarks (Liu et al., 2025; Zhou et al., 2026; Zhang et al., 2026) mainly compare models and model families, rather than how the gain of a fixed graph model changes with the text representation. We therefore think evaluation should report graph-model gains together with the text-only baseline and a baseline-adjusted measure, and should include a sweep over text representations.

Your comment helped us make the logic of our claims more rigorous, and we are grateful for it. We would be glad to answer any further questions. Thank you.

References

[a] Lee et al. "Feature Distribution on Graph Topology Mediates the Effect of Graph Convolution." ICML, 2024.

I appreciate the authors' rebuttal. However, I believe the uncovered flaw in the analysis methodology requires a major paper rewrite beyond what has already been done and beyond what is likely possible during a rebuttal. Thus, I maintain my score.

AuthorsOpenReview ↗

Same Results and Stronger Evidence After Controlling for the Baseline

Thank you for your response, and we understand your point. We agree that the original paper did not sufficiently separate the observed SAGE–MLP gain from baseline-dependent ceiling effects. This required us to refine how the gain is interpreted, but it did not change the benchmark, the measurements, or the experimental design, and the measured results in Figure 3 and Tables 5–10 remain the same. Sections 5.2, 7.2, and 8 now clarify what those results support, and the new controls test whether the observed patterns persist when the baseline is held fixed.

The original results remain valid, and the added experiments strengthen their logic

The definition and the measurements are the same as in the original submission. The homogeneous gain is still SAGE minus MLP on the same representation (Section 5.2). In response to your concern, the revision states what this quantity does not mean (Section 5.2), separates the two causes of the contraction (Section 7.2, "Why the gain contracts"), rewords the sentence you quoted (Section 8), and states the scope of TGA more precisely, as a label-free diagnostic within a graph, not as a better ranker than a fully labeled MLP. If the contraction were only a baseline effect, it would disappear once the baseline is controlled. As shown below, it does not.

The original results already show that the baseline does not explain the contraction alone. All numbers below come from the original runs (Tables 5–10 and their saved checkpoints), without retraining.

  • A baseline ceiling alone cannot explain negative gains. On CroVal, SAGE is below the MLP for 9 of the 14 text representations (Table 6), in all five seeds for each, and SAGE's best seed is below the MLP's worst.
  • The baseline-adjusted gain also falls. The share of remaining errors that SAGE corrects, (SAGE − MLP)/(100 − MLP), falls as MLP accuracy rises on all six HTAGs (Spearman −0.67 to −0.99).
  • A higher baseline can come with a larger gain. On TMDB, Sentence-T5 has a higher MLP than BGE-large (80.53 vs. 79.89) and a larger gain (+4.93 vs. +3.34). On CroVal, BGE-M3 has a higher MLP than TF–IDF (86.02 vs. 85.45) and a larger gain (+1.23 vs. +0.35). A ceiling alone predicts the opposite.
  • Matched baselines, different gains. Within a graph, representations whose MLP accuracy differs by at most one point differ in gain by up to 2.7 points (Appendix K.1).
  • The remaining errors are not harder for the graph model. On test nodes that both MLPs misclassify, strong-text SAGE is at least as accurate on all six HTAGs, using the original checkpoints (Figure 4a). Of the nodes that only the strong-text MLP solves, 49–73% were already solved by weak-text SAGE through the graph, so strong text now solves directly what the graph used to add.
  • Representation selection is not affected. These results compare absolute accuracies, not gains. The default features lose 2.50 (SAGE) and 2.64 (RSAGE) points to the best of 14 representations.

What the added controls show

With the MLP identical by construction, permuting features within (label, split) groups raised the gain in all 50 settings, and across six levels of this permutation TGA computed before any GNN run ordered the gain on all five tested HTAGs (median Spearman −0.86 to −1.00). With features fixed, rewiring citations changed the gain (Appendices K.2, K.6). These controls also show that TGA carries information beyond text-only accuracy, which the original correlations alone could not establish.

We designed the controls above around the concern that raw gains over different baselines are not directly comparable. Since these controls do not overturn the original results, we believe the original measurements and experimental setup are sound. The core question and the experimental results remain, while the revision refines how the gain is interpreted and the scope of our claims.

These results are already part of the revised paper. Section 7.2 summarizes them and points to Appendix K, so the revision as it stands reports the evidence that separates our findings from the baseline effect.

Taken together, these analyses do not make raw gains directly comparable across different baselines. They show that the baseline level alone does not explain the observed variation: gains also change when the MLP is held fixed, and the baseline-adjusted and same-error analyses support the same interpretation. We have revised the paper to state this distinction and to limit our claims accordingly, and in doing so we confirmed that the problem the paper addresses and its main findings did not change. In your review, you noted that the conclusions might be useful for the field if the analysis is correct, and we hope these analyses address that condition. If you have any further questions, please let us know. Thank you again for engaging with our response, and we would be grateful if you would reconsider your assessment!

Reviewer_gyD5OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how node representations affect the utility of graph structure in heterogeneous text-attributed graphs. It introduces MeHTAGs Bench, which pairs six datasets with 15 node representations while holding the graphs, labels, and data splits fixed. By comparing MLP, GraphSAGE, and relation-aware GraphSAGE, the authors report that the gains from homogeneous message passing generally decrease as text representations improve, whereas the additional gains from relation-aware modeling show no consistent trend. The paper also proposes Text–Graph Alignment (TGA), a label-free, training-free statistic that compares the similarity of meta-path-connected node pairs with that of random pairs. TGA is reported to correlate with residual homogeneous message-passing gains within each dataset. However, the study primarily varies input representations under fixed graph structures, leaving the broader question of when graph structure is needed insufficiently resolved.

Critical strengths
  1. Systematic experimental comparisons. The paper evaluates 15 node representations across six heterogeneous text-attributed graphs while keeping the graph structures, labels, and data splits fixed. It also examines the gains from homogeneous message passing and relation-aware modeling separately, showing how these gains change with the input representation.

  2. A label-free diagnostic. TGA measures text–graph alignment by comparing the similarity rankings of meta-path-connected node pairs and random pairs, without requiring labels or downstream model training. Across the six evaluated datasets, it shows a strong negative association with residual homogeneous message-passing gains.

Critical weaknesses
  1. The significance and scope of the research question are insufficiently justified. The experiments hold graph structure fixed and primarily vary node representations. They therefore establish how model gains change with input features on a given graph, but provide limited insight into when structural information itself is needed. The paper should better explain the practical importance of this feature-centered setting and the boundaries of its conclusions, particularly for graphs with weak or no node attributes.

  2. The evidence does not fully support the proposed use or interpretation of TGA. A correlation with the gain of SAGE over MLP does not establish that TGA can select the input representation with the best final classification performance. Furthermore, Table 4 disables aggregation using weights learned by a graph model, so it mainly measures how aggregation affects alignment rather than isolating the contribution of input features. An independently trained no-aggregation baseline, together with analyses of final representation class separability and classification performance, would help address these gaps.

Questions for authors
  1. Scope and significance. Since the graph structures are fixed throughout the representation sweep, what practical decision is this study intended to inform? Please clarify which conclusions concern the value of structural information itself and which concern the choice of input representation for an already specified graph.

  2. Use of TGA. Is TGA intended to help select an input representation for the best final graph-model performance, or only to rank the gain of SAGE over MLP? If representation selection is an intended use, how often would selecting by TGA identify, or come close to, the representation with the highest SAGE or RSAGE test performance in the existing experiments?

  3. Interpretation of Table 4. Given that the aggregation-disabled condition retains weights learned with aggregation enabled, what conclusion can be drawn about the contribution of the input features themselves? Can the authors compare this probe with the independently trained MLP results and report whether changes in TGA correspond to changes in final class separability and classification performance?

4 comments
AuthorsOpenReview ↗

Significance and Scope of the Research Question

Thank you for the careful review and the specific questions, which showed us where the motivation and scope were unclear. We revised the paper accordingly and respond in two parts. This first part addresses the significance and scope of the research question (Weakness 1, Q1).

Why study how the text representation changes what a graph adds?

1. Every HTAG benchmark fixes one text encoder, and its conclusions inherit that choice. Recent HTAG benchmarks (Liu et al., 2025; Zhou et al., 2026; Zhang et al., 2026) build node features with a single encoder and then compare graph models. A finding such as "relation-aware models help on this graph" is therefore a finding about one encoder–graph pair. For example, on the same graph, the gain of SAGE over an MLP moves by 6–9 points across encoders, and the default features lose on average 2.50 (SAGE) and 2.64 (RSAGE) points to the best of 14.

2. Encoders change faster than heterogeneous text-attributed graph benchmarks. Text embedding models have improved quickly (Enevoldsen et al., 2025; Lee et al., 2025; Zhang et al., 2025). From a result obtained with an older encoder, a practitioner can hardly tell what performance benefit a graph model offers on top of a current one. The best text-only representation is also not the best graph input on 3 of 6 datasets, so the encoder cannot be chosen from text-only accuracy.

3. Prior work shows that features can make structure unnecessary, but not when, or for which structure. MLPs distilled from GNNs can match them (Zhang et al., 2021). GNNs tend to overfit the graph and use it even when ignoring it would be better [a], which we also observe: with the strongest text on CroVal, SAGE falls 2.52 points below the MLP. For LLM-based predictors on text-attributed graphs, node text alone is already strong and most structural encodings add little (Xu et al., 2025), and LLM encoders outperform smaller language models most when the structure is less informative about the labels [b]. These studies, like text-attributed graph benchmarks (Yan et al., 2023), concern homogeneous graphs. They therefore cannot tell whether a gain comes from aggregating neighbors at all or from using relation types. This distinction matters for HTAGs, where simple homogeneous GNNs are known to rival heterogeneous ones (Lv et al., 2021).

4. However, existing diagnostics cannot answer this question. Homophily measures and label informativeness (Zhu et al., 2020; Ma et al., 2021; Platonov et al., 2023; Luan et al., 2023) depend only on the graph and its labels. Their values stay the same when the encoder changes, so they cannot explain why the gain changes when only the text representation changes.

What our study does

Building on this background, we evaluate six HTAGs with 15 representations while holding the graph, labels, and splits fixed, and we analyze the change in gain from the side of the representation. This leads to two claims.

  • When structure is needed. Neighbors help to the extent that they carry label information the node's own text lacks. We test this by changing the text–graph overlap and the structure directly, with text-only accuracy fixed (Part 2, Q1).
  • How to choose a representation. TGA shortlists representations before any graph model is trained and without labels. Its top four candidates recover the exhaustive RSAGE choice on all six datasets at 39% of the GPU cost.

Our conclusions therefore concern the choice of text representation on a given heterogeneous text-attributed graph and how much a graph model then adds. We hope this explains why we believe the question is worth studying. Part 2 addresses the weaknesses you raised and the experiments mentioned above.

References

[a] Bechler-Speicher, M. et al. "Graph Neural Networks Use Graphs When They Shouldn't." ICML, 2024.

[b] Wu, X. et al. "When Do LLMs Help With Node Classification? A Comprehensive Analysis." ICML, 2025.

AuthorsOpenReview ↗

When Structure Is Needed, the Use of TGA, and Table 4 (Q1–Q3)

This is Part 2, with the experiments for Q1–Q3. New results are in Appendix K (changes in blue).

Q1. When is structure needed, and which conclusions concern structure itself?

The decision is which text representation to feed a graph model on a given HTAG, and which candidates are worth training. Our results also give a condition for when structure helps: neighbors help to the extent that they carry label information the node's own text lacks. Features a node shares with its neighbors are redundant with its text. We tested both parts with text-only accuracy fixed (Appendix K.6). The first result concerns the choice of representation on a fixed graph, the next two the structure itself, and the last weak or missing attributes.

  • Less feature similarity, more gain. Permuting features within (label, split) groups leaves the MLP's data and accuracy unchanged. The gain of SAGEskip (SAGE with RSAGE's self skip) rose in all 50 dataset–representation settings on five HTAGs (+2.5 to +12.0 points on average).
  • Less label information, less gain. Rewiring citations lowered TGA and the gain together in all 8 settings on DBLP and arXiv. When the graph changes, TGA's label term moves, so TGA compares inputs on a fixed graph only.
  • Typed structure. Removing one relation at a time, the relation whose TGA comes more from feature similarity loses its value faster as text improves, on arXiv, CroVal, and DBLP. Removing the field-of-study relation cuts RSAGE's advantage over SAGEskip by 76–78%.
  • Weak attributes. With TF-IDF and GloVe, SAGE adds 6–12 points on five of six datasets. With random features, the value of structure depends on the graph (DBLP RSAGE +29.64, CroVal about 0).

Boundary. These conclusions hold within the evaluated graphs. Random features are not graphs that natively lack attributes, which we did not evaluate, and TGA is not comparable across graphs. The revised introduction and limitations state both.

Q2 (W2). Is TGA meant to select the best input, or only to rank the gain?

Short answer. TGA ranks the gain and also supports selection. Without labels, RSAGE trained on TGA's top four candidates recovers the exhaustive choice on all six datasets at 39% of the GPU cost. We do not claim it always picks the single best input.

What TGA measures. TGA is a label term plus a feature term, the similarity of connected nodes beyond shared labels (content term, Appendix K.6). A correlation alone cannot show which term matters, so we fixed MLP accuracy and varied this similarity at six levels. TGA, computed before any GNN run, ordered the gain on all five HTAGs:

TMDBCroValarXivBookDBLP
Median Spearman, TGA vs. gain−0.86−1.00−1.00−1.00−0.97
R², MLP accuracy alone0.710.420.400.560.22
R², with TGA added0.890.860.900.930.90

Selection. The top-TGA representation is the test-best on 3/6 datasets for both SAGE and RSAGE, and 11 of 12 choices are within 0.6 points. For RSAGE, TGA loses 0.42 points with no labels, against 0.59 for a fully labeled MLP, 1.61 for results borrowed from other HTAGs, and 2.64 for the default features. For SAGE, encoder size (0.03) and a fully labeled MLP (0.02) do better than TGA (0.13).

Screening. To match this, MLP accuracy needs five candidates, encoder size nine, and borrowed results eleven (Appendix K.3).

Scope. Across the evaluated encoders, a fully labeled MLP ranks gains better, and one with 0.1% of labels does so on 4 of 6 datasets. TGA needs neither labels nor training, and we see this as its strength (Section 7.3).

Q3 (W2). What does Table 4 show, given that its weights were trained with aggregation?

We agree that Table 4 cannot isolate what features alone can learn. Section 7.4 now presents it as a descriptive probe, and two analyses use independently trained models (Appendix K.4).

  • Separately trained MLP and SAGE (82 settings, same node pairs). Across representations, the change in TGA tracks the change in class separability (ρ = 0.56–0.95) and in Micro-F1 (0.56–0.98). Separability tracks accuracy (0.82–0.98), and higher input TGA predicts less added separability (−0.64 to −0.93).
  • Aggregation trained from scratch. We fixed the neighbor weight α ∈ {0, 0.25, 0.5, 0.75, 1} and trained SAGE at each α (TMDB, CroVal, arXiv, four representations, five seeds). α = 0 is the independently trained no-aggregation baseline you asked for. On all three datasets, the lowest-TGA input gains 8.9–11.5 points from aggregation and the highest −2.2 to +2.0. Final-representation TGA barely tracks separability on CroVal (0.23), so we use input TGA as the diagnostic.

Therefore, with independently trained models, inputs that are already aligned with the graph gain less from aggregation, in both separability and accuracy.

We hope the two parts address your concerns and are glad to answer more questions. Thank you.

Thank you for the detailed response and additional experiments. I consider my original second and third concerns largely resolved. Regarding the first concern, the authors have clarified that the study focuses on representation selection within a given heterogeneous text-attributed graph. The feature-shuffling, edge-rewiring, and relation-removal experiments further clarify feature–structure interactions and the boundaries of the conclusions. Nevertheless, I remain concerned about the significance of the research question and the extent of the contribution. The results primarily establish that input representations affect graph-model gains within the evaluated graphs and candidate text encoders, and that TGA can assist representation screening. While practically useful, these findings do not yet provide sufficiently deep and generalizable insights into when structural information is necessary or what complementary information different relations contribute. I do not expect the paper to cover all attribute-free graph settings, but even within its clarified scope, I remain insufficiently convinced that the contribution meets the acceptance threshold. So I can only raise my score to 2.

AuthorsOpenReview ↗

When Structure Is Needed, What Relations Contribute, and Limits on Generality

Thank you for reading both parts and for raising your score. We are glad that the concerns about TGA and Table 4 are largely resolved.

On your remaining point, we agree that showing that the input affects the gain would not be enough. We think the revision supports two statements that are conditions rather than trends over particular encoders, and we tested each with text-only accuracy held fixed.

When structure is needed

A graph helps through label evidence that a node's own text lacks. Features that a node shares with its neighbors add little once the text expresses them. Cutting this shared similarity at a fixed MLP raised the gain in all 50 settings. In evidence units, 61–78% of the decline in gain on CroVal, arXiv, DBLP, and Patent is evidence that the graph no longer adds (41–63% under a binary link). The rest is the stronger baseline. Where neighbors share little beyond labels (TMDB), the baseline explains the whole decline. On the evaluated graphs, better encoders reduced the value of structure where neighbors share features and not where they share little, and this can be measured before training.

What relations contribute

Each relation carries label evidence and feature similarity in different proportions, and we measure the feature-similarity share of each relation's TGA before training. On all three graphs we tested, the relation with the larger share loses its value faster as text improves. The other keeps a loss above noise at the strongest encoder on DBLP (+1.06 points) and narrowly on CroVal. Relation-specific weights pay off when relations differ in label evidence. Removing the field-of-study relation, whose edges agree on labels least, cuts RSAGE's advantage over SAGEskip by 76–78%. Corrupting citations hurts SAGEskip far more than RSAGE (−2.00 vs. −0.08 points on DBLP).

Limits on generality

These findings come from six graphs and fifteen representations. Public HTAG benchmarks that release raw text across several domains and scales are still limited (Liu et al., 2025; Zhou et al., 2026; Zhang et al., 2026), so we understand that generalizing from these results is difficult. Synthetic graphs from contextual stochastic block models (Deshpande et al., 2018) can vary feature sharing with vector features, but they cannot test text encoders, because realistic node text would also have to be generated. We therefore agree with your point on generality, and the limitations state it.

Your sharp comments helped us improve this work and bring together claims that had been spread across the appendix. Thank you again for the discussion!

Reviewer_EoBkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how the utility of graph structure varies with the choice of node text encoder. The authors evaluate 15 representations across six heterogeneous text-attributed graphs, keeping the graph structure, labels, and splits fixed throughout. They quantify the benefit of untyped message passing as the performance gap between SAGE and MLP, and the additional benefit of relation types as the gap between RSAGE and SAGE. Results show that the first gain generally diminishes as text representations improve, while the second shows no consistent trend. The proposed Text–Graph Alignment score measures the cosine similarity of meta-path-connected target nodes relative to random pairs. Higher TGA correlates with smaller SAGE-over-MLP gains within each graph.

Critical strengths
  1. The controlled experimental design is convincing. By testing 6 graphs, 14 text-derived representations plus a random control, 5 random seeds, and a broader set of architectures, the authors demonstrate that the observed representation-dependent patterns are robust and not merely an artifact of a specific MLP/SAGE comparison.
  2. TGA is interpretable and requires neither labels nor a trained classifier. The authors effectively rule out several obvious alternative explanations through comparisons with geometric and similarity statistics, width-controlled experiments with Qwen, and a fixed-weight aggregation probe.
Critical weaknesses
  1. The main contribution is more descriptive than explanatory. It is perhaps unsurprising that stronger text features leave less room for homogeneous message passing, especially since the gain is defined as SAGE minus MLP, and MLP performance itself correlates strongly with this gain in Table 3. The more novel and valuable finding is that relation-aware gains behave differently. However, the paper does not explain when or why relation types retain value as text quality improves. The fixed-weight aggregation probe in Table 4 captures shifts in alignment but does not pinpoint the source of downstream relation-aware gains. Despite the extensive sweep, the work reads more like a careful empirical report than a new theoretical account of graph utility.
  2. TGA has not yet been shown to improve practical decision-making. While its mean absolute Spearman correlation with homogeneous gain is 0.800, no experiment demonstrates that using TGA to select embeddings actually saves training time or leads to better final model selection. Appendix J acknowledges that TGA predicts neither the sign nor the magnitude of the gain. Additionally, the time-based protocol needs clarification: because the graph is fixed while labels are split temporally, it is unclear which edges are available to TGA and the GNN at each cutoff. This may be a valid transductive setting, but it limits how the diagnostic can be interpreted or deployed in practice.
Questions for authors
  1. How much of the decline in homogeneous gain persists after accounting for the stronger MLP baseline or matching text-only performance across representations? What controlled comparison could explain why relation-aware gains do not follow the same downward trend?
  2. If TGA were used to pre-select only a few candidates for graph-model training, how close would the resulting best model be to one found via exhaustive sweep, and how much computation would be saved? Please provide comparisons against random selection and text-only baselines.
  3. When computing TGA and auxiliary features for each time split, which edges and node texts are visible? Do the results change if these computations are restricted strictly to information available at the training cutoff?

I am willing to adjust my rating if these questions could be resolved.

4 comments
AuthorsOpenReview ↗

Why the Gain Contracts and When Relation Types Keep Their Value (W1, Q1)

Thank you for the constructive review and the clear questions. Each led to a new experiment. We respond in two parts. This part addresses Weakness 1 and Q1. New results are in Appendix K (changes in blue).

Q1a. Does the decline persist beyond the stronger MLP baseline?

Yes, at a baseline fixed by construction. We permuted target features only within (label, split) groups, the feature shuffle of Lee et al. [a], which cuts the feature similarity between connected nodes, leaves the MLP unchanged, and is known to raise GNN accuracy. We apply it across encoders. The gain of SAGEskip (SAGE with RSAGE's self skip) rose in all 50 dataset–representation settings on five HTAGs, by +2.5 to +12.0 points on average. The share of remaining errors it corrects falls with text strength on the original features and becomes nearly flat:

DatasetOriginal, weakest → strongestPermuted
TMDB27.9% → 21.5%33.8% → 32.2%
CroVal19.9% → 0.9%39.3% → 50.5%
arXiv33.6% → 11.8%57.9% → 60.0%
Book27.1% → 10.1%44.3% → 43.8%
DBLP28.3% → 9.6%58.5% → 57.9%

TGA orders the gain at a fixed baseline. We varied this similarity at six levels with identical MLP accuracy. TGA, computed before any GNN run, ordered the gain on all five HTAGs:

TMDBCroValarXivBookDBLP
Median Spearman, TGA vs. gain−0.86−1.00−1.00−1.00−0.97
R², MLP accuracy alone0.710.420.400.560.22
R², with TGA added0.890.860.900.930.90

Where our procedure raised the similarity (TMDB, Book), the gain fell (−1.26, −0.41 points).

Why the MLP correlates with the gain in Table 3. Within an HTAG, SAGE varies 1.5–6 times less than the MLP across representations, so the gain is close to a constant minus the MLP. This is why we tested the account with the MLP fixed.

How much is the baseline. In evidence units (probit link, Appendix K.6), a curve with constant graph evidence explains 22–39% of the observed decline on CroVal, arXiv, DBLP, and Patent (up to 59% under a binary link). The rest is evidence the graph no longer adds. On TMDB, where this similarity is weak, the baseline explains all of it. At matched MLP accuracy (70–76, 24 cells), the gain still ranges from +2.6 to +9.9 across graphs, and on errors both MLPs share, strong-text SAGE is at least as accurate on all six datasets (Figure 4a).

An account, not only a description. TGA decomposes exactly into a label term and a feature-similarity term (Proposition K.1). In a stylized model, a stronger encoder removes graph evidence only when neighbors share features with the node beyond their labels. The permutation removes this sharing, and the gain returns. TGA is thus related to the baseline but carries information that MLP accuracy lacks.

Q1b. When and why do relation types keep their value?

Part of the relation-aware gain was the self skip. RSAGE differs from SAGE by a self skip and relation-specific weights. Against SAGEskip (mean Micro-F1 points, six representations on Patent):

TMDBCroValBookarXivDBLPPatent
RSAGE − SAGE2.461.650.960.301.160.99
RSAGE − SAGEskip0.610.110.050.451.500.50

The gain largely disappears on TMDB, CroVal, and Book and persists on arXiv, DBLP, and Patent.

Where it persists, it depends on what each relation carries. A neighbor helps through label information that the node's own text lacks, and features it shares with the node are redundant. We removed one relation at a time (RSAGE, four representations, five seeds, shares computed before training):

DatasetRelation removed (feature-similarity share of TGA)Loss, GloVe → NV-Embed-v2
arXivauthor (0.56)+0.91 → +0.21
arXivfield of study (0.73)+2.58 → +1.11
CroValuser (0.67)+0.12 → +0.21
CroValtag (0.77)+2.98 → +0.01
DBLPauthor (0.75)+2.14 → +1.06
DBLPfield of study (0.83)+2.51 → +0.92

On all three graphs, the relation whose TGA comes more from feature similarity loses its value faster as text improves. Its larger initial loss may partly drive the faster fall. The other relation keeps a loss above noise under the strongest encoder on DBLP and narrowly on CroVal, but not on arXiv. Rewiring citations hurts SAGEskip far more than RSAGE (DBLP: −2.00 vs. −0.08), and removing the field-of-study relation cuts RSAGE's advantage by 76–78%, so separate weights help when relations differ in information. No entity relation (author, field of study, tag, user) decides the labels: a majority vote over the best one stays below the weakest text-only MLP (arXiv: 59.7 vs. 72.3). Unlike Table 4, these interventions measure downstream performance.

References

[a] Lee et al. "Feature Distribution on Graph Topology Mediates the Effect of Graph Convolution." ICML, 2024.

AuthorsOpenReview ↗

TGA Screening and Temporal Robustness (W2, Q2–Q3)

This is Part 2, on Weakness 2, Q2, and Q3.

Q2. How close is TGA screening to an exhaustive sweep, and what does it save?

Short answer: training RSAGE on TGA's top four candidates, instead of all 14, yields the same final model with 39% of the GPU-hours. TGA itself needs no labels and no training, so the only cost is training the graph model on the candidates it keeps. Under each rule, we train RSAGE on the top-k of 14 representations (12 on Patent) and select by validation. Loss is the mean test Micro-F1 gap to training all of them (exhaustive search, 222 GPU-h), and GPU-h is the RSAGE training time for the kept candidates.

RuleTop-2 loss / GPU-hTop-3 loss / GPU-hk to match exhaustive on 6/6
TGA (no labels)0.18 / 410.10 / 674 (86 GPU-h, 39%)
MLP accuracy (all labels)0.33 / 560.29 / 775 (111)
Encoder size0.40 / 550.29 / 769 (174)
Results borrowed from other HTAGs––11 (192)
Random (expectation)1.62 / 331.22 / 5014 (222)

TGA has the lowest loss at both k, and its top two beat the top three of the other rules. Computing TGA takes 0.8 CPU-hours for all six datasets, while MLP ranking needs all labels and 3.3 GPU-hours.

Single choice. If only one representation is trained, the top-TGA choice is the test-best on 3/6 datasets for both SAGE and RSAGE, and 11 of 12 choices are within 0.6 points. For RSAGE, TGA loses 0.42 points, against 0.59 for a fully labeled MLP, 1.61 for borrowed results, and 2.64 for the default features. For SAGE, encoder size (0.03) and a fully labeled MLP (0.02) do better than TGA (0.13).

What TGA does not do. As Appendix J states, it does not predict the sign or size of the gain across graphs, and across the evaluated encoders a fully labeled MLP ranks gains better. We see its strength in needing neither labels nor training. Section 7.3 now separates gain ranking, single choice, and screening.

Q3. What is visible under the temporal split, and do results hold at the training cutoff?

Original protocol. As the benchmark datasets used in our study follow a transductive setting, all node texts and edges were visible in our original protocol, with only labels split by time. TGA uses no labels and no auxiliary features. The GNN's auxiliary features were computed by averaging the features of all connected target nodes, including those from later periods.

Restricted to the training cutoff, the conclusions hold. We recomputed everything under this visibility:

StageStructureTarget-node textsLabels
TGA computationTraining-period graphTraining periodNone
GNN trainingTraining-period graphTraining periodTraining
Model selectionGraph up to validation periodUp to validation periodValidation
Test evaluationFull graphAll periodsTest
  • TGA. Its ranking is nearly the same as with the full graph (Spearman 0.978–0.996).
  • GNNs. SAGE and RSAGE trained and selected under this restriction, with auxiliary features recomputed per period, still show larger gains for weaker text in all 12 comparisons (six datasets, two models). The gains differ from the original by 0.30 points on average and by at most 2.15 points.

Because TGA needs no labels or training, it can be recomputed whenever the graph is updated. Appendix K.5 gives both protocols side by side.

These results therefore support a concrete use of TGA: screening representations without labels before graph-model training. In our sweep, its top four recover the exhaustive search’s selected RSAGE model on all six datasets using 39% of the RSAGE training GPU-hours. The temporal checks further show that TGA rankings remain nearly unchanged and the gain patterns persist when TGA computation and GNN training use only information available at the training cutoff. We hope these additions address your concerns about practical utility and the temporal protocol. We would be happy to answer any further questions. Thank you.

Thanks for the detailed rebuttal. The new experiments in Appendix K address my concerns to some extent. I am raising my rating from 2 to 3.

AuthorsOpenReview ↗

Thank You for the Discussion

Dear Reviewer EoBk,

Thank you very much for reviewing our response and for raising your rating before the deadline. Your questions led to the additional experiments in Appendix K, which we believe have made the claims of this work more robust. We are truly grateful for such a meaningful discussion. If any point remains unresolved, please let us know at any time, and we will respond before the discussion period ends. Thank you!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussion for the AC and SAC

Dear AC and SAC,

Thank you for handling our submission. We summarize the discussion below. Two reviewers raised their ratings following the rebuttal. The additional experiments are reported in Appendix K, and the main text has been revised to state our claims more precisely (changes in blue).

Our central claim is that the benefit of graph structure in an HTAG is a property of the graph–representation pair, not of the graph alone. A graph helps through label evidence that the node's own text lacks, so the same graph can help with one encoder and hurt with another. This makes evaluation across encoders necessary, and TGA offers a label-free way to screen representations before training.

Reviewers agreed on the strengths. EoBk found the controlled design convincing and TGA interpretable without labels or training. gyD5 highlighted the systematic comparison of 15 representations across six HTAGs and the label-free diagnostic. A7XS found the paper very well-written and the conclusions interesting and novel.

  • EoBk: raised. EoBk initially questioned baseline effects, why relation types retain value, TGA's practical utility, and temporal visibility. We added direct tests for all four points. EoBk acknowledged that the new experiments addressed the concerns to some extent and raised the rating.
  • gyD5: raised. gyD5 considered the concerns about TGA and Table 4 largely resolved after the representation-screening experiments and independently trained no-aggregation baselines. Reservations about significance and generality remain. We clarified the practical decision our study informs and the scope of the conclusions, including the difficulty of generalizing from the limited set of available HTAG datasets.
  • A7XS: maintained. A7XS questioned comparisons of gains over different MLP baselines. We agree that raw gains alone cannot separate baseline effects from feature–structure interactions, and we added controls that hold MLP accuracy identical by construction: permuting features within (label, split) groups increased the gain in all 50 tested settings (Appendix K.6). A7XS maintained the rating, citing the extent of rewriting the concern would require, but did not question the new results themselves. In our reply, we noted that the measured results are unchanged and that the revision refines their interpretation.

Some concerns arose because the original text did not state the scope of our claims clearly enough, and others called for controls the original paper lacked. The reviewers' comments helped us address both, and none of the added controls reversed a main finding. The main pattern holds with the text-only baseline fixed by construction, across five seeds and the architecture sweep, and when TGA and GNN training use only training-period information (Appendix K.5). We therefore believe the results can be useful to the community, as A7XS noted they would be if the analysis is correct. The revised evidence supports three contributions.

  • TGA supports representation screening before graph-model training. TGA ranks candidate representations without labels or downstream training. Training RSAGE on its top four candidates recovers the exhaustive sweep's selected model on all six datasets with 39% of the GPU-hours, while MLP accuracy needs five candidates and encoder size nine (Appendix K.3). This selection result is distinct from gain ranking, where a fully labeled MLP can perform better.
  • Relation-aware gains reflect both architecture and the information each relation carries. After matching the self skip, RSAGE's advantage largely disappears on TMDB, CroVal, and Book but persists on arXiv, DBLP, and Patent. On the three graphs tested with relation removal, the relation with the larger feature-similarity share of TGA loses more value as text improves. Together with citation rewiring, these results indicate that relation-specific weights help when relations differ in label information (Appendices K.2, K.6).
  • HTAG evaluation should use several text encoders. On CroVal, SAGE adds 4.87 points over the MLP with DistilBERT features and loses 2.52 points with NV-Embed-v2 (Appendix K.1), and the best text-only representation is not always the best graph input. A benchmark that fixes one encoder can therefore reach opposite conclusions about the same graph. We recommend reporting gains with the text-only baseline over a set of encoders fixed in advance that varies along several criteria, such as encoder family, encoder size, and TGA, rather than a subset chosen after seeing results. TGA can be one of these criteria. It is computed before any training and without labels, so it cannot be tuned to test results, and because higher TGA goes with smaller homogeneous gains, it does not favor inputs that inflate the reported value of the graph.

We respectfully ask that the submission be assessed on this evidence and its stated limitations. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.