acceptodds
Under review as a conference paper at ICLR 2027

Teaching LLMs to See Graphs: Unifying Text and Structural Reasoning

Abstract

Applying Large Language Models (LLMs) to graph-structured data usually involves multi-step pipelines in which textual node attributes are compressed into single tokens and further processed by GNNs, often discarding most of their semantic content. We introduce the Graph Transformer Language Model (GTLM), which enables a pretrained LLM to process graph topology natively and removes this bottleneck entirely. GTLM injects graph-aware attention biases directly into the LLM's attention modules, adding only 0.015% structure-related parameters relative to the base model. Training updates only the structural parameters together with a LoRA adapter on the base model. We prove that our bidirectional attention prefix is permutation-equivariant over nodes and that GTLM reduces exactly to the pretrained model when no graph is present. Having no global node ordering, GTLM shows no positional degradation and does not *get lost in the middle*: needle-in-a-graph accuracy stays flat from 1k to 64k tokens and 4× past the training length, while an identically trained flat-text LLM baseline collapses. Comprehensive evaluations show that GTLM matches or exceeds domain-specific state-of-the-art models on text-attributed graph benchmarks, GraphRAG on WebQSP, and molecular benchmarks, while meaningfully improving over strong baselines on GraphQA. We further show that GTLM's attention heads implicitly learn to simulate message passing, explaining its strength on algorithmic tasks. Together, these results suggest that a minimally adapted pretrained LLM can serve as a general backbone for graph learning.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
LLMs, Graph Machine Learning, Graph Transformers

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4BXUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GTLM, which adds graph-derived attention biases to pretrained LLMs so they can use node text and connectivity together. It combines per-node positional resets with bidirectional prefix attention to remove dependence on the order in which context nodes are presented.

Critical strengths
  1. GTLM keeps token-level representations of node text and adds graph structure directly to attention. The design for node permutation equivariance is also clearly motivated.
  2. Experiments cover several graph-related tasks. The synthetic retrieval comparison, using the same backbone and training budget, gives convincing evidence of gains in multi-hop retrieval and length generalization.
Critical weaknesses
  1. The added benefit over an information equivalent, same backbone, same budget text LoRA baseline is not well established. Removing biases in Appendix K can also remove connectivity information, and the effects of biases, positional resets, and prefix attention are not fully separated.
  2. The paper includes 1B, 3B, and 8B experiments, but mainly relies on Llama3. BLOOM and Gemma are tested only on GraphQA. Whether the gains hold on more recent representative backbones remains unclear.
  3. Separate fine-tuning shows within task effectiveness, but does not tell us whether a trained adapter can be reused on new tasks.
  4. Recovering standard computation for a single node does not undo LoRA updates. The paper does not evaluate how general capabilities change after graph-task training.
Questions for authors
  1. With connectivity explicitly described in the text, how does GTLM compare with a standard LoRA tuned LLM using the same backbone and training budget? Separately, can the contributions of structural biases, positional resets, and bidirectional prefix attention be disentangled?
  2. Do these relative gains reproduce on more recent representative backbones, such as Qwen?
  3. Can a trained adapter transfer to unseen tasks without target task fine-tuning?
  4. With the graph trained LoRA enabled, how do general capabilities compare with those of the original model?
Reviewer_ADnWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GTLM, which encodes graph topology using shortest-path distances, relative random-walk probabilities, and Magnetic Laplacian features, and injects these structural signals into a pretrained LLM through learnable attention biases. By preserving node-level text tokens and jointly modeling textual content and graph structure, GTLM enables graph-conditioned reasoning and text generation across tasks such as node classification, graph question answering, and molecular understanding.

Critical strengths
  1. Overall, this paper is well written and easy to follow.
  2. Enabling LLMs to natively understand graphs is an interesting direction, and the proposed architecture makes sense to me. Injecting node-level structural biases into attention while keeping the full node text avoids the compression bottleneck of soft-token methods.
  3. The design is simple and lightweight.
Critical weaknesses
  1. The paper claims that injecting graph structure into attention is better than serializing graphs into text. However, a plain-text fine-tuning baseline with the same backbone is only provided for the synthetic needle-in-a-graph task. Adding it on main benchmarks such as Cora, WebQSP and ChEBI-20 would help show whether the gains come from the structural attention or simply from fine-tuning on the full node texts.

  2. Using the Magnetic Laplacian bias to capture edge direction is an interesting design. However, its only ablation is conducted on GraphQA, where direction plays a limited role, so it is not yet fully clear whether the bias actually captures edge direction or mainly contributes general spectral information. I think the paper could be further strengthened by a simple experiment in which the model enumerates the in- and out-neighbors of a node across different graph sizes and node degrees. This would nicely show how reliably direction is captured and how this ability scales.

  3. Similar work on enabling a pretrained LLM to natively process graphs was published earlier this year and is worth mentioning, specifically "Weaving Graph over Tokens: Contextualizing Structured Sequences for LLMs" [1], which has similar properties and ideas.

[1] Jiaxuan Chen, Zixing Zhang, Ruijun Mao, Wei Sun, Zhicheng Liang, Yuhang Zhang, Yaxi Liu, and Fangxin Wang. Weaving Graph over Tokens: Contextualizing Structured Sequences for LLMs. ICML 2026.

Questions for authors
  1. For disconnected node pairs, the RRWP and Magnetic features are all zeros and the SPD is beyond the maximum distance, so all such pairs seem to receive the same bias. How does the model distinguish between different disconnected nodes, for example, an isolated question node versus an irrelevant component?
Reviewer_RTxeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes GTLM, which integrates graph structural information directly into pretrained LLMs through learnable attention biases, combined with node-wise positional encoding and bidirectional prefix attention. Experiments across various graph-related tasks demonstrate promising results.

Critical strengths
  1. The proposed architecture is simple and intuitive, enabling LLMs to jointly process textual and structural information without compressing node attributes into graph tokens.

  2. The paper provides theoretical analysis of permutation equivariance and backward compatibility.

  3. The experiments cover a diverse range of graph-related tasks and demonstrate competitive performance.

Critical weaknesses
  1. Missing closely related work. My primary concern is that the paper overlooks several highly relevant studies, including Graph Language Models (ACL 2024), Graph-KV (NeurIPS 2025), and GaLA (KDD 2026). These works have already explored incorporating graph structural information directly into language-model attention and addressing positional biases caused by graph serialization. The authors should clarify how GTLM fundamentally differs from these approaches and better establish its methodological novelty.

  2. Insufficient experimental comparisons. Related to the above concern, the experiments mainly compare GTLM against conventional GNNs, graph-token-based LLMs, and standard LLMs, while omitting closely related graph-aware attention approaches. This makes it difficult to assess whether GTLM offers meaningful improvements over existing methods. Comparisons with Graph-KV and GaLA, where applicable, would substantially strengthen the empirical evaluation.

  3. Limited analysis of the architectural contributions. GTLM simultaneously introduces graph attention biases, bidirectional prefix attention, and node-wise positional encoding. However, the long-context experiments primarily compare against a standard LLM, making it unclear which components drive the substantial improvements. More controlled ablations would help isolate their individual contributions.

Questions for authors
  1. How does GTLM differ from closely related approaches such as Graph Language Models (ACL 2024), Graph-KV (NeurIPS 2025), and GaLA (KDD 2026)? Could the authors clarify the key methodological novelty compared to these works?

  2. Why are these closely related graph-aware LLM approaches not included in the experimental comparisons? Could the authors provide additional comparisons or explain why direct comparisons are not feasible?

  3. To what extent do the improvements in long-context graph reasoning come from the proposed structural attention biases, rather than bidirectional prefix attention or node-wise positional encoding? Could the authors provide more controlled ablation studies to disentangle their contributions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.