Learning Graph Representations via Predictive Language Supervision

Graph–language pretraining typically relies on autoregressive token prediction or graph–text alignment. Token prediction ties supervision to a particular linguistic realization, while alignment may suppress information that is unnecessary for matching but useful for downstream graph tasks. We introduce GL-JEPA, which uses a frozen Text Encoder to define a continuous target for each description and a Predictor to reconstruct this target from graph states. Predicting the full description latent reduces dependence on surface form while encouraging graph states to capture description information that may otherwise be suppressed by alignment. For downstream adaptation, query conditioning allows the Predictor to extract task-relevant information from frozen graph representations. Compared with token prediction, GL-JEPA improves by 25.2–43.8 percentage points the rate at which two valid descriptions both outrank a negative. In a controlled diagnostic, latent prediction achieves 95.6% probe accuracy on a target-relevant factor, versus 12.7% for alignment with easy negatives. GL-JEPA also outperforms direct alignment on node-, link-, and graph-level tasks and supports query-conditioned graph question answering across nine tasks with a frozen Graph Encoder.