acceptodds
Under review as a conference paper at ICLR 2027

GraphToken: A General Graph Vocabulary for Text-attributed Graph Foundation Models

Abstract

Graph foundation models (GFMs) aim to enable generalizable graph learning across diverse tasks and domains, yet constructing transferable representation units for text-attributed graphs (TAGs) remains challenging due to the coupling of graph structure and textual semantics. Existing graph tokenization methods often derive tokens primarily from structural patterns or rely on continuous graph embeddings, leaving the learning of a shared discrete vocabulary over structure–text patterns underexplored. To address this challenge, we propose **GraphToken**, a motif-based discrete vocabulary learning framework for TAGs. For each target node, GraphToken samples multiple small rooted motif instances from its local neighborhood and jointly encodes their topology and node texts into root-conditioned motif representations. These representations are vector-quantized against a shared learnable codebook, where each codebook entry corresponds to a reusable **graph-token** shared by structurally and semantically related instances. A target node is consequently represented by a common set of graph-tokens that characterize its local structure–text context. Importantly, GraphToken decouples vocabulary induction from downstream prediction, allowing the learned tokens to be used by Graph Neural Networks (GNNs), Transformers, and Large Language Model (LLM)-based approaches alike, regardless of their architectural differences. Extensive experiments across graph domains, tasks, and model architectures demonstrate the effectiveness and generalizability of the learned GraphToken vocabulary.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph Foundation Models, Text-Attributed Graphs, Graph Tokenization, Graph Vocabulary

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3kR2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes GraphToken, a method that turns the local neighbourhood of each node in a text-attributed graph into a set of discrete tokens. For each node, it samples 32 small connected subgraphs ("rooted motifs") of two to four nodes, encodes each of them with a GIN over the text embeddings of their nodes, and assigns each encoding to the nearest entry of a learnable codebook, as in vector quantization. The node is then represented by the resulting multiset of codebook entries ("graph-tokens"), which can be fed to a GNN (after mean pooling), a Transformer, or an LLM through a learnable projector. The evaluation covers node classification on ten datasets from citation, e-commerce, and social domains, link prediction on three datasets, transfer between datasets, an ablation of text and structure, and an analysis of individual tokens. The LLM variant reaches the best or close to the best results on most datasets.

Critical strengths

The idea of a shared discrete vocabulary over joint structure-text patterns is natural for text-attributed graphs, and separating this vocabulary from the downstream model is a reasonable design goal. The evaluation is broad (10 datasets from three domains and 15 baselines), and the analysis of individual tokens in Figure 4 and Appendix A.6 is a useful attempt to show what the vocabulary captures.

Critical weaknesses

1. The method cannot be reproduced from the paper or from the released code.

  • The paper never states how the motif encoder and the codebook are trained. It only says that they "are jointly optimized through vector quantization" (lines 256-257), but vector quantization alone does not define a learning objective.
  • Key settings are missing: the codebook size KK, the text encoder ftextf_{\text{text}}, and, for the LLM variant, the prompts, how the projector is trained, and whether the LLM is frozen (Figure 2 marks the LLM variant as learnable).
  • The paper states that the "implementation and configuration files are available" (lines 512-513), but the repository contains only model components, without training or evaluation scripts, configurations, or data loading, and its README contains only a title. The LLM variant, which gives the main results, is missing: llm.py contains only text-encoding helpers and a head that compares node embeddings with label text embeddings (llm.py:59-68). Moreover, transformer.py implements attention between neighbouring nodes of the graph (transformer.py:30-48), while Eq. 6 describes attention among the graph-tokens of one node. Finally, the only loss in the code is a term that pulls the motif encodings toward their nearest codebook entries (quantizer.py:74-77), which confirms that the training objective is defined elsewhere and is not released.
  • Much of the text uses abstract and decorated wording instead of simple statements, which hides these missing details. For example, "GraphToken establishes a shared interface between discrete graph-tokens and downstream architectures. For each target node, its local context is represented by a collection of recurring graph-tokens, yielding a vocabulary-grounded characterization of the neighborhood" (lines 78-81) means only that each node is represented by the tokens of its sampled motifs, and these tokens can be used as input to different models.

2. The experiments do not support the main claims, and several results look wrong.

The experiments do not show that the discrete vocabulary is a key component:

  • There is no ablation without quantization (using the continuous motif encodings), and no comparison of the same backbone with and without GraphToken on the same text features. Therefore, the gains may come from the text encoder or from Vicuna-7B rather than from the vocabulary.
  • Only the LLM variant is competitive. The GNN and Transformer variants are often far below the baselines, for example, 35.21 and 38.06 macro-F1 on arXiv23 against 56.40 for RGLM (Table 1). This weakens the claim that one vocabulary serves different backbones equally well.
  • The margins of the LLM variant are small and often within one standard deviation of the best baseline, for example, 75.46 vs 75.28 accuracy on arXiv and 67.82 vs 67.65 on Reddit, and GOFA is better on PubMed (92.84 vs 92.36).
  • The analysis sections do not support the claims drawn from them. The "most transferable tokens" are ranked by an ad-hoc score with four loosely defined, equally weighted terms (Appendix A.4.2), and removing them changes the results by about 2 percentage points, for a transfer to Reddit that, according to Table 3, barely works (see below). The sampling analysis (Figures 5-6) measures only the speed and the number of sampled subgraphs, and never shows whether the sampler affects accuracy.

Moreover, several reported results look wrong or come from a flawed setup:

  • In Table 3, all methods reach 5.6-9.4% accuracy on Cora when trained on S2, although Cora has seven classes, so random guessing gives about 14%, and a prediction head is trained on the Cora training split (lines 915-917). Similarly, on the binary Reddit task with S1, the baselines reach 44-51% accuracy and GraphToken 52%, although always predicting the majority class gives more than 50%. Both results suggest an error in the setup.
  • In Table 3, the source collections S1 and S3 contain the target datasets Cora and History, so these settings do not measure transfer, although the text describes them as sources that contain "graphs from the target domain" (lines 377 and 400).
  • In Table 1, the GRAVER results on Fitness (52.83 ± 3.03 and 36.43 ± 2.30) are identical to its results on arXiv, which suggests a copy error.

3. The paper overlooks closely related work, which also limits its novelty.

  • Section 2 presents graph foundation models as if they were limited to text-attributed graphs. However, many real-world graphs have numerical or categorical node features without any text, and an active line of work builds foundation models that handle such arbitrary feature spaces, for example, TAG [1], G2T-FM [2], and GraphPFN [3], which build on tabular foundation models and prior-data fitted networks. These methods pursue the same goal of transfer across graphs, and some of them already show practical gains on diverse real-world data, yet the paper does not mention any of them.
  • Quantizing GNN encodings of local subgraphs against a learned codebook was already done by GFT, which builds a tree vocabulary over text-attributed graphs, as well as by VQGraph [4] and GQT [5], which also trains its tokenizer separately from the downstream model. The paper describes GFT only as a structural approach and does not cite the other two. As a result, the main difference from prior work appears to be the use of sampled rooted motifs instead of computation trees, and the paper does not test whether this choice matters.

[1] Hayler et al. "Bringing Graphs to the Table: Zero-shot Node Classification via Tabular Foundation Models," 2025

[2] Eremeev et al. "Turning Tabular Foundation Models into Graph Foundation Models," 2025

[3] Eremeev et al. "GraphPFN: A Prior-Data Fitted Graph Foundation Model," 2025

[4] Yang et al. "VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs," 2024

[5] Wang et al. "Learning Graph Quantized Tokenizers," 2025

Questions for authors
  1. What is the full training objective of the motif encoder and the codebook?
  2. What are the codebook size, the text encoder, and the training procedure of the LLM variant?
  3. How do the results change without quantization, and how does a GNN or Vicuna-7B with the same text features but without GraphToken perform? How do they change when GraphToken uses random-walk or computation-tree sampling instead of rooted motifs?
  4. Why is the accuracy on Cora and Reddit in Table 3 lower than random guessing, and how were the foundation-model baselines trained on the target datasets?
Reviewer_3X5VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GraphToken, which encodes sampled rooted subgraphs from text-attributed graphs into joint structural and textual representations, and then constructs a shared discrete vocabulary through vector quantization. Each node is represented by multiple motif tokens, which are integrated into GNNs, Transformers, and LLMs through different interfaces. The authors evaluate the method on node classification, link prediction, and vocabulary transfer tasks across multiple domains, reporting competitive results. Ablation studies further investigate the contributions of combining structural and textual information.

Critical strengths
  1. Using rooted subgraphs as the basic units for joint structural and textual representations is intuitive and well-motivated. This design explicitly preserves local connectivity and the structural role of the target node, while organizing different instances through a shared codebook. It provides a meaningful approach to learning a graph vocabulary.

  2. The experiments cover multiple domains, both node classification and link prediction, and three types of downstream architectures. The overall results are competitive, with particularly noticeable improvements on Cora. The ablations involving structure, text, and random pairing also provide evidence for the value of properly combining the two types of information.

Critical weaknesses
  1. Some key training details remain unclear. The code provides some details of codebook learning, but the authors need further clarify how the tokens are adapted to different downstream models.

  2. The existing ablations are insufficient to isolate the contribution of the discrete vocabulary. While they support the benefits of combining structural and textual information, a direct comparison against continuous motif representations is missing. Given the existence of related work on shared graph vocabularies, such a comparison would be important for better understanding the specific contribution of this paper.

Questions for authors
  1. Could the authors provide more details about the training procedures for the codebook and the LLM projector?

  2. What would happen if the subgraph representations were passed directly from the encoder to the same downstream models, without being replaced by representative vectors from the codebook? Such a comparison would help clarify the contribution of the discrete vocabulary.

  3. How exactly is the "content-independent text" constructed in the structure-only setting? The paper interprets its low codebook utilization as "severe vocabulary collapse." However, if all nodes are assigned the same substitute features, the number of distinguishable rooted subgraph patterns may already be quite limited. Could the authors clarify why this observation should be interpreted as vocabulary collapse?

Reviewer_e1LKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a graph foundation model, called GraphToken, which is a motif based discrete vocabulary learning framework for text attributed graphs. For each target node, GraphToken samples small rooted motifs togather with their node texts and uses them to maintain a shared learnable codebook. This design enables GraphToken to transfer accross different downstream tasks and architectures like GNNs, Transformers, and LLMs. Several experiments are conducted to demonstrate the effectiveness of proposed method.

Critical strengths
  • S1: The core idea has good potential in scalability as it only samples small local motifs and maintains a small vocabulary codebook.
  • S2: The experiments cover a wide range of datasets and baselines which makes the empirical results convincing, although more comprehensive settings should be included.
Critical weaknesses
  • W1: My main concern is the novelty. To my best knowledge, the vector quantized tree vocabulary is used by GFT [1], the motif pattern encoding is used by GPM [2], which makes the GraphToken look like a recombination of previous techniques.
  • W2: The transferability is one of the core advantage of GraphToken, however, the transfer learning experiments focus on node classification mostly. For a recent graph foundation model, it would be expected to enable more open-ended tasks such as QA and graph-to-text generation

[1] GFT: Graph Foundation Model with Transferable Tree Vocabulary. 2024

[2] Generative Graph Pattern Machine. 2026

Questions for authors
  • Q1: Why do some transfer learning settings in table 3 use the same datasets for both source and target graphs?
  • Q2: Can you highlight the novel contributions of GraphToken against previous works? If the work is essentially an orchestration of existing methods, can you analyze why this design or pipeline works better?
  • Q3: For ablation study in figure 3, apart from the current results, can you show that the text-only variant actually encodes semantic information, and structure-only variant encodes topological information? For example, you can mask some features of the training data to see if these approachs achieve similar performance.
  • Q4: I am curious if the graph vocabulary is directly usable by LLMs

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your thoughtful reviews and contributions. The authors have now submitted their rebuttals. Please take some time to review their responses and assess whether your concerns have been adequately addressed. If appropriate, please update your comments and ratings accordingly.

Thank you for your time and continued efforts.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.