acceptodds
Under review as a conference paper at ICLR 2027

Frozen Text Embeddings with a GNN is “All You Need” for Text Classification

Abstract

Recent work in text classification constructs token-level graphs from the embeddings of a frozen language model and trains a graph neural network (GNN) on these graphs to perform downstream classification. We show that existing approaches lack consistent performance across standard benchmarks, with no single graph construction strategy dominating across datasets. To address this limitation, we introduce HILT-GCN, a simple hierarchical token-level graph architecture that models multi-layer interactions via layer-specific virtual hubs at linear complexity - HILT-GCN achieves the strongest and most stable overall performance among frozen token-level approaches across five single-label benchmarks. Compared with strong fully fine-tuned baselines based on both small and large text encoders, HILT-GCN achieves competitive results on four out of five datasets, falling behind only on Ohsumed. By using frozen embeddings, HILT-GCN supports any new text backbone without encoder fine-tuning, achieving high accuracy while substantially reducing the parameter and training overhead of large-scale models.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
text classification, graph neural networks, token-level graphs, frozen language models, multi-layer representations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gMCwOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies text classification with embeddings from a frozen LLM and a GNN head built on token-level graphs. It abstracts recent methods into a general pipeline, consisting of five stages. It then proposes HILT-GCN, which creates one node per token per layer, adds one virtual hub per layer connected to all tokens of that layer, and uses a global root connected to all hubs. Experiments cover five benchmarks and six backbones, using five different seeds. The authors report that HILT-GCN is the best frozen method overall and achieves results close to fine-tuned embedders. They also report ablation results to test different components.

Critical strengths

All mentioned methods in paper are implemented in a unified framework, and evaluated across different backbones, datasets, and seeds. It then identifies existing methods that lack consistent performance across different benchmarks, with none consistently outperforming the others across all datasets. Moreover, It proposes a simple method HILT-GCN, which models token interactions in linear complexity and is easy to implement, achieving the strongest results across multiple benchmarks.

Critical weaknesses
  1. Virtual nodes are a standard GNN technique, and exploiting multi-layer transformer representations for probing is well established, HILT-GCN seems like a combination of these ideas, and do not claim in the related-work section.

  2. In the constructed graph, token nodes of different layers are connected via token_1 → hub_1 → root → hub_2 → token_2, a path of length 4. So there is no information from other layers that reaches token nodes for K less than 4, and the root node has no effect on the token-node representations used for readout. The default is set to 2 in this paper, which means that cross-layer information is combined in the final mean‖max pooling, not through message passing on the graph.

  3. There is no ablation that keeps all-layer token nodes, the same projection, the mean‖max readout, and the MLP head but removes the edges (or removes only the root node). The main comparison in Table 1 is not controlled for head and readout: the baselines use a linear head with mean (or attentional) pooling, while HILT-GCN uses a 2-layer MLP with LN plus mean‖max pooling.

  4. Table 3 states that variants “differ solely in the graph construction mechanism.” If the TokenGraph, GLOT, and AttnGraph constructions still use only final-layer tokens (as in Table 5), the comparison mixes the effect of topology with the effect of using all layers.

  5. If stability refers to seed variance, HILT-GCN’s standard deviations are often larger than the baselines.

  6. All five datasets are mostly topic classification, it should include more diverse tasks.

  7. The method is called “Hierarchical GCN” in the Figure 3 legend but HILT-GCN in the text.

Questions for authors
  1. Why do the HILT-GCN numbers in Table 3 differ from those in Table 1, if both use the same model and GNN backbone?

  2. How do the re-implemented baselines compare with the results reported in their original papers on overlapping datasets?

2 comments

We thank Reviewer gMCw for the detailed review.

W1: Virtual nodes and multi-layer probing [...]

HILT-GCN is motivated as a scalable extension of token-graph probing baselines rather than claiming virtual nodes or multi-layer probing in isolation. Section 3 cites Ulanovski et al. (2026) to motivate all-layer token graphs alongside GNN routing, and Appendix E cites Tenney et al. (2019). In Section 7, we will add cross-references to Section 3 and cite foundational virtual-node GNNs (Gilmer et al., 2017; Ishiguro et al., 2019) and multi-layer probing works.

W2: Cross-layer token paths have length 4 [...]

This observation is correct and matches Section 3 (~Line 154), default K=2K=2 captures localized token transformations (k=1k=1) and intra-layer hub context (k=2k=2). Since virtual nodes are filtered out at Stage 4, K=2K=2 routing is strictly intra-layer (token->hub->token), while cross-layer fusion occurs in the global readout. Cross-layer routing activates at K=4K=4 (Table 4) with minor differences; thus we use K=2K=2 to match baselines and will clarify that vrootv^{\text{root}} only participates for K≥4K\ge4.

W3 & W4: Edge removal, readout/head, and single-layer ablations

Table 1 primarily compares the baseline methods as documented in the literature, showing that a relatively simple architecture can achieve competitive results. Table 3 addresses confounding GNN and readout factors, showing that a simple multi-layer topology is sufficient for improving evaluation. Full multi-layer token availability is not isolated here, as prior baselines operate only on final-layer tokens or layer averages.

To isolate edge removals, head & readout, as well as restricting full-layer access, we implement the requested controls (reporting Micro F1/Macro F1).

HILT-GCN Variant20NGR8R52OhsumedMR
Default90.05/89.3898.39/95.3596.75/81.4578.08/69.8291.55/91.54
w/o Res & JK89.21/88.5198.55/95.8696.88/84.2680.45/73.1191.49/91.49
Graph w/o Edges90.21/89.5698.25/95.3397.01/84.6879.50/71.6791.45/91.45
Mean Pool + Linear--97.51/92.5394.49/73.0771.02/63.4189.58/89.56
Macro-F1 Early Stop90.47/89.8598.36/95.4496.87/85.7178.80/72.0491.05/91.05
Last Layer89.52/88.8197.98/94.3896.25/83.7676.40/68.8390.86/90.85
  1. Readout & Head (W3): Mean||Max + MLP head drives major gains; Mean Pool + Linear drops sharply, matching Table 4.
  2. Edges vs. Skip Connections (W3): Per Section 6 (Table 4), combining hub routing with Res+JK skip connections adds optimization overhead. Removing either edges (Graph w/o Edges) or skip connections (w/o Res & JK) alleviates this: Graph w/o Edges leads on 20NG/R52, simplified hub routing (w/o Res & JK) leads on R8/Ohsumed (+2.37 Micro/+3.29 Macro-F1 over Default), and Default leads on MR. We will add these controls to Section 6 to credit multi-layer Mean||Max readout alongside simplified hub routing (see Reviewer gam6, W1 for further non-graph controls).
  3. Last vs. All Layers (W4): Restricting HILT-GCN to the final layer drops scores across all five benchmarks toward last-layer baselines, confirming multi-layer features drive primary gains.

W5: Seed standard deviations.

“Stable” in the Abstract and Section 5 refers to cross-dataset and cross-backbone consistency, not seed variance. We will revise this to “consistent across benchmarks and backbones.”

W6: Task diversity beyond topic classification.

We focus on categorical text classification for single-label, supporting multi-label settings; other tasks such as NLI and QA are left as future work. You may refer to our response to Reviewer YVHX showcasing promising results for multi-label evaluation.

W7: Figure 3 legend naming.

Thank you for catching this; we will update the Figure 3 legend to HILT-GCN (Ours).

Q1: Why do HILT-GCN numbers in Table 3 differ from Table 1?

Per Appendix A.3, Table 1 ran across A100/H100 nodes while Table 3 ran on a single H100 node. Non-deterministic sparse CUDA atomics (scatter_add in PyG) yield minor variations within std dev, except Macro-F1 on the imbalanced R52 dataset. Adjusting Early Stopping to Macro-F1 (HILT-GCN (Macro-F1 Early Stop)) stabilizes this difference.

Q2: Comparison of re-implemented baselines with original papers.

Among the 4 graph baselines, GLOT/CayleyGCN share no datasets with our benchmarks, and TokenGraph overlaps only on MR strictly in a few-shot regime (confirmed in direct exchange with the authors). AttnGraph has the only full-dataset overlap (R8, Ohsumed). On bert-base-uncased, our re-implementation closely matches their R8 Accuracy (97.41±0.34 vs. 97.85±0.25) and is lower on Ohsumed (67.15±0.39 vs. 69.96±0.51). While exact factors are hard to isolate across codebases, this gap likely stems from dataset splits (their 20% val vs. our 10% TextGCN split); notably, AttnGraph remains the strongest frozen baseline on BERT Ohsumed in Table 1.

Thank you for the detailed response and the additional experiments. The clarifications on W1, W5, W6, W7, Q1 and Q2 address my concerns.
For W2, the authors confirm that with K=2 the root node does not participate and cross-layer fusion happens only in the readout, which means the claimed hierarchical cross-layer interaction does not take place under the default setting. For W3 & W4, the results confirm my concerns.
First, w/o Res & JK appears in both Table 4 and this response, but the results differ noticeably, especially on 20NG, while the default results in this response are identical to Table 1. All variants should be re-run under the same setup and seeds.

Second, w/o Res & JK performs better than the default on three of five datasets and has a higher average in Table 4, suggesting that the residual connections and JK in the proposed design are unnecessary. In addition, Graph w/o Edges outperforms the default on three of five datasets, and the non-graph control (in the response to Reviewer gam6), the Learned Mixture of Layer-Pooled Features, performs comparably to HILT-GCN without any token-level graph. Together, these results indicate that the gains should come from multi-layer features, mean‖max pooling, and the MLP head rather than from the proposed hierarchical graph.

In summary, I will keep my rating.

Reviewer_YVHXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a token-level graph-learning method called HILT-GCN. It utilizes a frozen PLM encoder to tokenize texts, then it uses well-designed graph constructor, GNN module, and graph readout to produce a representation embedding for downstream tasks. HILT-GCN achieves competitive performance on single-lable text classification against baseline methods accross various benchmark datasets. The proposed method also doesn't require fully fine-tuning for different target graphs.

Critical strengths
  • S1: The experimental results on single-label text classification is strong.
  • S2: The proposed method shows good efficiency in computation and compatibility with different PLMs.
  • S3: The writing is clear and easy to follow.
Critical weaknesses
  • W1: Lack of evidence in generalization capability to tasks other than single-label text classification. Since graph representation learning methods already have demonstrated ability to downstream tasks such as retrieval and semantic similarity as shown in GLOT, only single-label text classification limits the scope of the proposed method.
  • W2: Incomplete ablation studies. For example, the GCN module can be replaced with more recent GNN architectures. Currently it's only hyperparameter tuning of GCNs.
  • W3: Missing references. For example, RG-Hybrid: pure late fusion of RoBERTa and a graph transformer for robust, interpretable sentiment classification.
Questions for authors
  • Q1: I am curious about how does the jumping knowledge mechanism work under the hood? Do some virtual hubs have significantly higher weights than others? Can you visualize this with a heatmap?
  • Q2: As the dicision models are emerging, how can HILT-GCN stay advantageous against these highly efficient and general-purposed model? I believe the reuseable representations need to be explored more, not only for classification and regression tasks. For example, QA and capturing structural relation within a sentence can be alternative tasks to explore. Can the output of HILT-GCN support these type of tasks?
  • Q3: Can we apply HILT-GCN to STS tasks by concatenating two sentences?
2 comments

We thank Reviewer YVHX for the constructive feedback. We address each comment and question below.

W1: Lack of evidence in generalization capability to tasks other than single-label text classification.

We focus on categorical text classification for single-label, supporting multi-label settings; other tasks such as NLI and QA are left as future work. To demonstrate that our pipeline generalizes beyond single-label classification, we replace the Stage 5 softmax CE loss with sigmoid BCE loss without adjusting any other part of the architecture (reporting Micro-F1 / Macro-F1):

ArchitectureWOSRCV1-V2NYT
SFT Runs (Ceiling — bert-base-uncased)
Linear (bert-base-uncased)86.87 / 81.3486.24 / 67.7279.61 / 69.03
MLP (bert-base-uncased)86.70 / 81.0186.26 / 67.0479.40 / 68.30
Frozen Models (Qwen3-Embedding-8B)
Linear73.11 / 63.6883.24 / 62.7673.58 / 60.24
MLP74.60 / 65.8484.54 / 63.7574.76 / 62.75
TokenGraph83.75 / 76.3586.31 / 65.5376.92 / 64.75
GLOT85.59 / 79.2085.41 / 65.2376.89 / 65.36
CayleyGCN75.09 / 65.8185.15 / 64.6077.05 / 63.85
HILT-GCN (Ours)87.70 / 82.3186.75 / 67.3179.72 / 69.14

Note that these multi-label datasets have hierarchical label structures that specialized in-domain methods typically exploit. However, even treating them as a flat label set yields strong results. AttnGraph is omitted due to the prohibitive runtime of extracting multi-layer attention matrices from Qwen3 on larger corpora.

W2: Incomplete ablation studies. [...] more recent GNN architectures [...]

We agree that more elaborate GNN architectures could be explored. We focus on GCNs primarily because they keep the architecture simple and already achieve competitive results, aligning with recent graph learning findings ([1] Sancak, Balin, Çatalyürek. "Do We Really Need Complicated Graph Learning Models? - A Simple but Effective Baseline." LoG, 2024.; [2] Luo, Shi, Wu. "Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence." ICML, 2025.)

Additionally, our baselines already employ more expressive, attention-based GNN architectures; specifically GAT (TokenGraph, GLOT) and multi-head GATv2 with edge attributes (AttnGraph) (Table 5). For further control ablations, we refer to our response W1 from Reviewer gam6.

W3: Missing references. For example, RG-Hybrid [...]

We thank you for bringing this approach to our attention and will cite and discuss it in Section 7. While RG-Hybrid is conceptually similar, it is not directly comparable within our experimental setup because it relies on partial fine-tuning. A core constraint across all of our baselines is using the underlying language models strictly as frozen feature extractors, as partially unfreezing transformer layers introduces significant task-adaptation gains that would confound a frozen-backbone comparison (see, e.g., Lee et al., "What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning," CoRR abs/1911.03090, 2019).

Q1: I am curious about how does the jumping knowledge mechanism work under the hood? [...]

Virtual hubs are connected via unweighted edges and used strictly for message-passing. In Stage 4, virtual hubs are filtered out prior to readout. JK is implemented via feature concatenation across propagation hops. Because w/o Res & JK (Table 4) shows that this mechanism can be omitted without hurting performance, explicit hop weighting is unnecessary.

Q2: As the dicision models are emerging, how can HILT-GCN stay advantageous against these highly efficient and general-purposed model? [...] Can the output of HILT-GCN support these type of tasks?

Regarding efficiency the primary computational bottleneck is the single-pass transformer activation for both HILT-GCN, as well as decision models. Even without caching and further optimizations (Table 2), the latency for one sample is around 40 ms, remaining certainly competitive with API-bound models like Jev (~70ms-500ms; https://typesafe.ai/blog/introducing-system-one-models-and-jev - 2026-11-13).

HILT-GCN can also be modified to accommodate other tasks. It provides representations at both the document level (the graph embedding zGz_\mathcal{G} supports sentence-pair or multiple-choice QA) and the token/node level (the modulated node feature matrix ZJKZ_\text{JK} supports extractive QA and structural relations) using lightweight task-specific heads (Stage 5).

Q3: Can we apply HILT-GCN to STS tasks by concatenating two sentences?

Consistent with the previous answer, this also applies to STS. Concatenating two sentences works out of the box, though building two separate graphs for both sentences could also be beneficial.

Thank you for your responses and efforts. The additional experiments and clarifications have addressed many of my concerns, especially those weaknesses. However, I still have reservation on the broader value, since Q2 and Q3 still lack convincing experiment designs and results to support the application of HILT-GCN beyond text classification. Therefore, I maintain my score.

Reviewer_gam6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates text classification using frozen transformer representations and proposes HILT-GCN, a hierarchical graph classifier over token embeddings from multiple transformer layers. Each layer has a virtual hub connected to its token nodes, and a global root connects the layer hubs. A residual GCN with Jumping Knowledge processes this graph, followed by mean–max pooling over token nodes and an MLP classifier.

The evaluation covers five single-label classification datasets and six transformer backbones. HILT-GCN generally improves over the evaluated frozen-feature baselines and approaches fine-tuned performance on several datasets. The paper also reports training throughput, memory consumption, and architectural ablations.

Critical strengths
  1. Practical motivation and simple design. Improving downstream classification without updating the transformer is useful, particularly when the backbone is large or shared across tasks. The proposed graph has a straightforward construction with a linear number of nodes and edges.

  2. Substantial empirical coverage. The study includes five datasets, six backbones, frozen and fine-tuned comparisons, and five-run mean and standard deviation reporting. Reporting both Micro-F1 and Macro-F1 is also valuable.

Critical weaknesses
  1. The contribution of graph structure is confounded with richer input features and pooling.

The Linear and MLP baselines use only a pooled representation from the final transformer layer. Several graph baselines also use final-layer token features, whereas HILT-GCN retains all layer–token representations and applies mean–max pooling followed by a nonlinear classifier. Consequently, these comparisons change several factors simultaneously: feature availability, pooling, graph structure, and classification head. Keeping the GNN module fixed in Table 3 is helpful, but it does not fully isolate topology when the graph constructions retain different node representations.

The most informative missing controls are a shared token-wise MLP applied to the same multi-layer tensor followed by the same mean–max pooling and classifier, a learned mixture of layer-wise pooled features, and HILT-GCN restricted to the final layer or a subset of layers. These comparisons would establish whether graph propagation contributes beyond better use of frozen features.

  1. The broad performance claims need qualification.

The results support HILT-GCN as a competitive frozen-feature classifier, but the claim of approaching fine-tuning on four of five datasets depends primarily on Micro-F1. For example, with Qwen3, R52 Macro-F1 is 81.45 for HILT-GCN versus 86.67 for the PEFT MLP, despite relatively close Micro-F1. On Ohsumed, the corresponding Macro-F1 scores are 69.82 and 76.47. HILT-GCN also does not consistently outperform the strongest frozen baseline on Macro-F1.

Five-run reporting is appreciated, but small margins should be supported with paired uncertainty estimates before being described as consistent superiority. The title and conclusions should reflect the demonstrated setting: five English single-label benchmarks with inputs truncated to 512 tokens. A longer-context scaling experiment or an additional contemporary classification benchmark would strengthen broader claims, but careful scoping is the immediate requirement.

Questions for authors

How does HILT-GCN compare with a non-graph model using the same multi-layer features, projection, pooling, and classifier?

2 comments

We thank Reviewer gam6 for the constructive feedback. We address each comment question below.

W1: The contribution of graph structure is confounded with richer input features and pooling.

Table 1 primarily compares the baseline methods as documented in the literature, showing that a relatively simple architecture can achieve competitive results. Table 3 addresses confounding GNN and readout factors, showing that a simple multi-layer topology is sufficient for improving evaluation. Full multi-layer token availability is not isolated here, as prior baselines operate only on final-layer tokens or layer averages.

To isolate the remaining factors, we evaluate the requested controls on Qwen3. Full bert-base-uncased equivalents will be included in the revised manuscript.

Architecture (Qwen3-Embedding-8B)20NGR8R52OhsumedMR
HILT-GCN Variants
HILT-GCN (Default)90.05 / 89.3898.39 / 95.3596.75 / 81.4578.08 / 69.8291.55 / 91.54
HILT-GCN (w/o Res & JK)89.21 / 88.5198.55 / 95.8696.88 / 84.2680.45 / 73.1191.49 / 91.49
HILT-GCN (Graph w/o Edges)90.21 / 89.5698.25 / 95.3397.01 / 84.6879.50 / 71.6791.45 / 91.45
Feature & Non-Graph Controls
Token-Wise MLP (All Layers)90.48 / 89.7898.46 / 95.4596.69 / 82.3180.60 / 72.8190.79 / 90.78
Learned Mixture of Layer-Pooled Features90.15 / 89.5298.15 / 95.1096.74 / 81.0978.84 / 71.1890.88 / 90.87
HILT-GCN (Last Layer)89.52 / 88.8197.98 / 94.3896.25 / 83.7676.40 / 68.8390.86 / 90.85
Macro-F1 Optimization
HILT-GCN (Macro-F1 Early Stopping)90.47 / 89.8598.36 / 95.4496.87 / 85.7178.80 / 72.0491.05 / 91.05

The results establish the following findings:

  1. Multi-layer token representations and Mean-Max readout drive the primary gains: Restricting HILT-GCN to the final layer drops scores across all five benchmarks towards the last-layer baselines, whereas multi-layer variants, including the Learned Mixture of pooled features, consistently improve performance.
  2. Edge-free controls and simplified hub routing both improve over Default HILT-GCN: As noted in Section 6 (Table 4), combining hub message passing with both Residual and Jumping Knowledge skip connections introduces optimization overhead. Removing either the edges (Graph w/o Edges, Token-Wise MLP) or the skip connections (HILT-GCN w/o Res & JK) alleviates this issue and leaves the two approaches closely matched: Token-Wise MLP slightly leads on 20NG and Ohsumed Micro-F1, Graph w/o Edges leads on R52, and pure hub routing (HILT-GCN w/o Res & JK) leads on R8 and Ohsumed Macro-F1 (while Default HILT-GCN leads on MR).

We will add these controls to Section 6 to explicitly credit the multi-layer Mean-Max readout alongside simplified graph routing.

W2: The broad performance claims need qualification.

The biggest gap between frozen and fine-tuned approaches is observable in Macro-F1 scores, especially for unbalanced label sets. This observation shows that reporting only Accuracy (Micro-F1) is insufficient. As Early Stopping is calibrated on the unweighted validation softmax CE loss, there is certainly a bias towards frequent classes, prioritizing Micro-F1 performance (Section 5, Line 307). Simply changing the Early Stopping criterion to Macro-F1 helps closing this gap (HILT-GCN (Macro-F1 Early Stopping) Variant).

In Section 5, we will revise claims on small margins, where standard deviations overlap, to “competitive/on par”. We will also soften the title, abstract, and conclusion to remove the “All You Need” phrasing, and state the 512-token truncation scope in accordance with the single-label nature of our benchmarks.

Q1: How does HILT-GCN compare with a non-graph model using the same multi-layer features, projection, pooling, and classifier?

The Token-Wise MLP (All Layers) baseline in the table under W1 above shows the results. It achieves very competitive performance, leading on 20NG and Ohsumed Micro-F1 while trailing on MR.

Thank you for the additional controls and proposed revisions. The token-wise MLP and edge-free variants address the missing comparisons, but their competitive or superior performance leaves the graph-specific contribution insufficiently supported beyond multi-layer feature access and pooling. The simplified graph variant shows some advantages, although their reliability remains unclear without uncertainty estimates. Narrowing the claims and reporting Macro-F1-based early stopping are welcome; however, comparable model-selection criteria should also be applied to the non-graph controls. Overall, the response improves transparency but does not sufficiently substantiate the architecture-centered contribution. I therefore maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.