- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies text classification with embeddings from a frozen LLM and a GNN head built on token-level graphs. It abstracts recent methods into a general pipeline, consisting of five stages. It then proposes HILT-GCN, which creates one node per token per layer, adds one virtual hub per layer connected to all tokens of that layer, and uses a global root connected to all hubs. Experiments cover five benchmarks and six backbones, using five different seeds. The authors report that HILT-GCN is the best frozen method overall and achieves results close to fine-tuned embedders. They also report ablation results to test different components.
Critical strengths
All mentioned methods in paper are implemented in a unified framework, and evaluated across different backbones, datasets, and seeds. It then identifies existing methods that lack consistent performance across different benchmarks, with none consistently outperforming the others across all datasets. Moreover, It proposes a simple method HILT-GCN, which models token interactions in linear complexity and is easy to implement, achieving the strongest results across multiple benchmarks.
Critical weaknesses
-
Virtual nodes are a standard GNN technique, and exploiting multi-layer transformer representations for probing is well established, HILT-GCN seems like a combination of these ideas, and do not claim in the related-work section.
-
In the constructed graph, token nodes of different layers are connected via token_1 → hub_1 → root → hub_2 → token_2, a path of length 4. So there is no information from other layers that reaches token nodes for K less than 4, and the root node has no effect on the token-node representations used for readout. The default is set to 2 in this paper, which means that cross-layer information is combined in the final mean‖max pooling, not through message passing on the graph.
-
There is no ablation that keeps all-layer token nodes, the same projection, the mean‖max readout, and the MLP head but removes the edges (or removes only the root node). The main comparison in Table 1 is not controlled for head and readout: the baselines use a linear head with mean (or attentional) pooling, while HILT-GCN uses a 2-layer MLP with LN plus mean‖max pooling.
-
Table 3 states that variants “differ solely in the graph construction mechanism.” If the TokenGraph, GLOT, and AttnGraph constructions still use only final-layer tokens (as in Table 5), the comparison mixes the effect of topology with the effect of using all layers.
-
If stability refers to seed variance, HILT-GCN’s standard deviations are often larger than the baselines.
-
All five datasets are mostly topic classification, it should include more diverse tasks.
-
The method is called “Hierarchical GCN” in the Figure 3 legend but HILT-GCN in the text.
Questions for authors
-
Why do the HILT-GCN numbers in Table 3 differ from those in Table 1, if both use the same model and GNN backbone?
-
How do the re-implemented baselines compare with the results reported in their original papers on overlapping datasets?
2 comments
Thank you for the detailed response and the additional experiments. The clarifications on W1, W5, W6, W7, Q1 and Q2 address my concerns.
For W2, the authors confirm that with K=2 the root node does not participate and cross-layer fusion happens only in the readout, which means the claimed hierarchical cross-layer interaction does not take place under the default setting.
For W3 & W4, the results confirm my concerns.
First, w/o Res & JK appears in both Table 4 and this response, but the results differ noticeably, especially on 20NG, while the default results in this response are identical to Table 1. All variants should be re-run under the same setup and seeds.
Second, w/o Res & JK performs better than the default on three of five datasets and has a higher average in Table 4, suggesting that the residual connections and JK in the proposed design are unnecessary. In addition, Graph w/o Edges outperforms the default on three of five datasets, and the non-graph control (in the response to Reviewer gam6), the Learned Mixture of Layer-Pooled Features, performs comparably to HILT-GCN without any token-level graph. Together, these results indicate that the gains should come from multi-layer features, mean‖max pooling, and the MLP head rather than from the proposed hierarchical graph.
In summary, I will keep my rating.
We thank Reviewer gMCw for the detailed review.
HILT-GCN is motivated as a scalable extension of token-graph probing baselines rather than claiming virtual nodes or multi-layer probing in isolation. Section 3 cites Ulanovski et al. (2026) to motivate all-layer token graphs alongside GNN routing, and Appendix E cites Tenney et al. (2019). In Section 7, we will add cross-references to Section 3 and cite foundational virtual-node GNNs (Gilmer et al., 2017; Ishiguro et al., 2019) and multi-layer probing works.
This observation is correct and matches Section 3 (~Line 154), default K=2 captures localized token transformations (k=1) and intra-layer hub context (k=2). Since virtual nodes are filtered out at Stage 4, K=2 routing is strictly intra-layer (
token->hub->token), while cross-layer fusion occurs in the global readout. Cross-layer routing activates at K=4 (Table 4) with minor differences; thus we use K=2 to match baselines and will clarify that vroot only participates for K≥4.Table 1 primarily compares the baseline methods as documented in the literature, showing that a relatively simple architecture can achieve competitive results. Table 3 addresses confounding GNN and readout factors, showing that a simple multi-layer topology is sufficient for improving evaluation. Full multi-layer token availability is not isolated here, as prior baselines operate only on final-layer tokens or layer averages.
To isolate edge removals, head & readout, as well as restricting full-layer access, we implement the requested controls (reporting Micro F1/Macro F1).
Mean||Max+ MLP head drives major gains;Mean Pool + Lineardrops sharply, matching Table 4.Graph w/o Edges) or skip connections (w/o Res & JK) alleviates this:Graph w/o Edgesleads on 20NG/R52, simplified hub routing (w/o Res & JK) leads on R8/Ohsumed (+2.37 Micro/+3.29 Macro-F1 over Default), andDefaultleads on MR. We will add these controls to Section 6 to credit multi-layerMean||Maxreadout alongside simplified hub routing (see Reviewer gam6, W1 for further non-graph controls).“Stable” in the Abstract and Section 5 refers to cross-dataset and cross-backbone consistency, not seed variance. We will revise this to “consistent across benchmarks and backbones.”
We focus on categorical text classification for single-label, supporting multi-label settings; other tasks such as NLI and QA are left as future work. You may refer to our response to Reviewer YVHX showcasing promising results for multi-label evaluation.
Thank you for catching this; we will update the Figure 3 legend to
HILT-GCN (Ours).Per Appendix A.3, Table 1 ran across A100/H100 nodes while Table 3 ran on a single H100 node. Non-deterministic sparse CUDA atomics (
scatter_addin PyG) yield minor variations within std dev, except Macro-F1 on the imbalanced R52 dataset. Adjusting Early Stopping to Macro-F1 (HILT-GCN (Macro-F1 Early Stop)) stabilizes this difference.Among the 4 graph baselines, GLOT/CayleyGCN share no datasets with our benchmarks, and TokenGraph overlaps only on MR strictly in a few-shot regime (confirmed in direct exchange with the authors). AttnGraph has the only full-dataset overlap (R8, Ohsumed). On
bert-base-uncased, our re-implementation closely matches their R8 Accuracy (97.41±0.34 vs. 97.85±0.25) and is lower on Ohsumed (67.15±0.39 vs. 69.96±0.51). While exact factors are hard to isolate across codebases, this gap likely stems from dataset splits (their 20% val vs. our 10% TextGCN split); notably, AttnGraph remains the strongest frozen baseline on BERT Ohsumed in Table 1.