acceptodds
Under review as a conference paper at ICLR 2027

Learning Graph Representations via Predictive Language Supervision

Abstract

Graph–language pretraining typically relies on autoregressive token prediction or graph–text alignment. Token prediction ties supervision to a particular linguistic realization, while alignment may suppress information that is unnecessary for matching but useful for downstream graph tasks. We introduce GL-JEPA, which uses a frozen Text Encoder to define a continuous target for each description and a Predictor to reconstruct this target from graph states. Predicting the full description latent reduces dependence on surface form while encouraging graph states to capture description information that may otherwise be suppressed by alignment. For downstream adaptation, query conditioning allows the Predictor to extract task-relevant information from frozen graph representations. Compared with token prediction, GL-JEPA improves by 25.2–43.8 percentage points the rate at which two valid descriptions both outrank a negative. In a controlled diagnostic, latent prediction achieves 95.6% probe accuracy on a target-relevant factor, versus 12.7% for alignment with easy negatives. GL-JEPA also outperforms direct alignment on node-, link-, and graph-level tasks and supports query-conditioned graph question answering across nine tasks with a frozen Graph Encoder.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Graph Language Model, Latent Predictive Learning, Zero-shot, Graph Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XbvXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors identify two potential limitations of existing graph–language pretraining approaches. Autoregressive token prediction requires reconstructing a particular linguistic realization of a graph description, while direct graph–text alignment may suppress information that is unnecessary for distinguishing matched graph–text pairs but remains useful for downstream graph tasks. To address these issues, GL-JEPA uses a frozen Text Encoder to obtain continuous representations of paired graph descriptions. A Graph Encoder processes graph structure and node-text attributes, while a trainable Predictor estimates the corresponding description representation. Training minimizes a normalized prediction loss without autoregressive token reconstruction or contrastive negatives. In the controlled comparison, GL-JEPA outperforms direct Alignment on node classification, link prediction, and graph classification, while Alignment achieves higher graph–text retrieval performance. A nearly parameter-matched Linear Predictor variant also performs better than Alignment on the three graph prediction tasks.

Critical strengths

Strength 1

The paper addresses a meaningful problem in graph representation learning: how to use paired natural-language descriptions as supervision without requiring exact token reconstruction or relying exclusively on contrastive graph–text matching. An important strength is the controlled comparison in Table 1. GL-JEPA improves over Alignment on node classification, link prediction, and graph classification. Although the full GL-JEPA Predictor has substantially more parameters than Alignment, the authors also evaluate a nearly parameter-matched Linear Predictor. This smaller variant retains advantages over Alignment on all three graph prediction tasks, suggesting that the improvements are not explained solely by the larger Transformer Predictor. 

Strength 2

The complete-system frozen-transfer results show that GL-JEPA performs competitively across heterogeneous benchmarks, achieving the highest mean scores among the evaluated methods on node, link, and graph classification. In the synthetic experiment, Alignment with easy negatives achieves approximately 12.7% probe accuracy on the target-relevant factor, while GL-JEPA reaches 95.6%. The WikiBio diagnostic shows a similar qualitative pattern. The combination of controlled diagnostics, frozen-transfer benchmarks, and component ablations is a substantial strength of the paper.

Critical weaknesses

Weakness 1

A central motivation of the paper is that direct graph–text alignment may suppress information that is unnecessary for distinguishing matched from mismatched pairs but remains useful for downstream tasks. However, the results demonstrate that the extent of feature suppression depends strongly on the construction of contrastive negatives. In the synthetic experiment, easy-negative Alignment achieves 12.7% probe accuracy on factor B, compared with 95.6% for GL-JEPA. However, hard-negative Alignment achieves 99.9%, exceeding GL-JEPA. A similar pattern appears in the WikiBio experiment, where hard-negative Alignment achieves 100% occupation probe accuracy, compared with 77.1% for GL-JEPA. The results suggest that feature suppression is not an inherent limitation of graph–text alignment, but can arise when the contrastive objective is satisfied using shortcuts provided by the negative-sampling strategy. The current evidence supports a narrower conclusion: predicting the full description target can preserve additional factors without requiring task-specific hard-negative construction under the examined settings. It remains unclear whether the same degree of feature suppression occurs in realistic graph–language pretraining with larger representations and more diverse negative sampling. 

Weakness 2

The graph question-answering results in Table 3 are presented as evidence that GL-JEPA learns graph representations that support competitive graph reasoning. But, the evaluation compares GL-JEPA with systems that do not necessarily use the same answer-prediction protocol. As described in Section 3.5, GL-JEPA performs discriminative candidate-answer scoring using a query-conditioned Predictor and a frozen answer Text Encoder. In contrast, several of the compared methods use generative language-model-based answering, potentially involving different output constraints, decoding procedures, and answer formats. This distinction is very important because Table 3 reports very large advantages on some benchmarks, including 99.48% accuracy on NLGraph Cycle. Without harmonizing the output protocols, it is difficult to determine how much of these improvements comes from the quality of the learned graph representations rather than differences in the answer interface. Furthermore, although the Graph Encoder remains frozen, the query-conditioned Predictor and output projection are adapted using task-specific supervision. The results therefore demonstrate the usefulness of the frozen graph representations under a trainable readout, rather than zero-shot reasoning by an entirely frozen model.

Questions for authors

Q1

The feature-suppression experiments show that hard-negative Alignment substantially improves factor recovery and even outperforms GL-JEPA in both diagnostic settings. Could the authors clarify what advantage latent prediction retains when contrastive alignment uses sufficiently informative negatives? 

Q2

Section 3.5 and Appendix E describe GL-JEPA as a discriminative model that scores candidate answers through a query-conditioned Predictor rather than generating free-form text. Could the authors clarify whether the baselines in Table 3 use equivalent candidate-answer constraints and evaluation procedures? How would the comparison change if all methods were evaluated using the same binary answer-scoring protocol? 

Q3

Section 2.3 states that the Predictor assigns consecutive rotary positions to graph-derived node tokens. Since node ordering is generally arbitrary in graph representations, does the resulting Predictor preserve permutation invariance? Could the authors report whether permuting node IDs while preserving node features and graph connectivity changes the predicted description representation or downstream performance? 

Q4

The method relies on a frozen Text Encoder to provide continuous supervision for graph representations. But, the target embedding may not preserve every semantic or structural detail contained in the original description. What evidence supports the assumption that predicting the full text embedding preserves a broader range of graph-relevant information than direct alignment?

5 comments

Thank you for the questions. Changes for this review are in green in the revised paper, and changes shared with Reviewer 8wpE are in purple.


Response to W1 and Q1: What does latent prediction offer when negatives are informative?

Latent prediction has no negatives to design, so no factor must be chosen for protection in advance. A contrastive loss keeps what separates positives from negatives and can suppress the rest [1, 2, 3], so harder negatives recover some features and suppress others [4]. In WikiBio, hard-negative Alignment reaches 100% on occupation but is at chance on nationality, which its candidates share (Figure 4, Table R3). With uniform negatives (β = 0), both objectives keep both factors, with Alignment 2.1 points higher on each. As more negatives share the anchor's nationality, Alignment loses nationality (35.4% at β = 2, chance from β = 4), while latent prediction keeps both without tuning. Section 3.3 states this narrower conclusion.

WikiBio probe (%, chance 25)OccupationNationality
Alignment, easy negatives27.1100.0
Alignment, hard negatives100.025.0
Latent prediction77.179.2

In the structural diagnostic, Alignment with random negatives or one hard negative per anchor leaves structure near chance up to width 8 (19.5% and 20.6%, chance 12.5%) and recovers it only with structure-defined positives, while latent prediction recovers it at every width (Table R2). Latent prediction can lose a factor when the graph state is small. The 128-way A (chance 0.8%) stays near chance up to width 8 and returns at width 32 (96.7%). In the Table 1 checkpoints (full corpus, 384-dimensional Graph Encoder), GL-JEPA graph states predict the number of nodes and the root degree better than Alignment on all seven datasets (Table R2(c)), and GL-JEPA is higher on three of the four transfer tasks (Table 1).

Structure probe (%, chance 12.5)Width 2Width 4Width 8
Alignment, easy negatives16.014.919.5
Alignment, one hard negative per anchor16.216.420.6
Alignment, structure-defined positives79.270.695.1
Latent prediction99.0100.0100.0

[1] Chen, 2021, Intriguing Properties of Contrastive Losses

[2] Li, 2023, Addressing Feature Suppression in Unsupervised Visual Representations

[3] Bleeker, 2023, Reducing Predictive Feature Suppression in Resource-Constrained Contrastive Image-Caption Retrieval

[4] Robinson, 2021, Can Contrastive Learning Avoid Shortcut Solutions?

Response to W2 and Q2: What changes under a common answer scoring?

In the original Table 3, three baselines were scored on two candidate answers like GL-JEPA, and the other four on generated answers. In the revision, every model is adapted, so the comparison is not zero-shot, and is scored by the likelihood of two candidate answers ({yes, no}, or {support, counter} on ExplaGraphs). Training settings follow each model's official script where available, and GL-JEPA adapts only the Predictor and output projection with the Graph Encoder frozen. Changed values are shown in purple in Table 3, and Appendix E gives the settings and data splits (Tables R6 and R7).

Accuracy (%). Bold marks the best value per benchmark. On NLGraph, 52.9 (Cyc.) and 54.2 (Conn.) come from answering yes to every question (Appendix E).

ModelTrainableSizeGraphInstructExplaGraphsNLGraph Conn.NLGraph Cyc.
GL-JEPA754M1B79.861.086.399.5
GraphGPT40.0M (LoRA)7B75.255.149.955.0
LLaGA2.0M (projector)7B62.451.350.151.3
GraphSolver335.5M (LoRA)8B54.958.764.252.9
TEA-GLM134.8M (graph module)7B52.759.054.252.9
GOFA1,006.9M (GNN layers)7B73.357.054.252.9
G1-3B239.5M (LoRA)3B81.893.398.490.6
GraphWiz6,738.4M (full, by authors)7B59.178.067.452.9

Response to Q3: Does permuting node IDs change the results?

The Predictor is not exactly permutation invariant, but node order has a small effect. Structure enters through the order-independent Graph Encoder, and order enters only through the rotary positions of node tokens, which leave node states unchanged. Permuting node IDs five times on Cora and CiteSeer with the Table 1 checkpoint keeps the mean cosine between predicted descriptions at 0.997 or higher and moves zero-shot node accuracy by at most 0.23 points on average, one or two test nodes (Table R4).


Response to Q4: What supports that predicting the frozen target preserves more than alignment?

Both objectives use the same frozen target, so information missing from the target is missing for both. Our claim concerns factors that are in the target but not needed for matching. Regression must fit every target coordinate, while a contrastive loss gives little supervision to factors that do not separate positives from negatives. In WikiBio, the target encodes occupation at 100%. Easy-negative Alignment lowers it to 27.1% (chance 25.0%), below the 64.6% of the untrained graph state, and latent prediction raises it to 77.1% (Figure 4, Table R3). Structural factors behave the same way (Table R2). The Limitations section discusses the limits of the target.

Dear Reviewer XbvX,

Thank you again for your questions. Before the discussion period ends, we would appreciate knowing whether our response answers them. In the WikiBio sweep, Alignment loses nationality as more negatives share it, while latent prediction keeps both factors without tuning (Table R3), and all QA baselines are now scored on two candidate answers (Table 3). We would be glad to discuss any remaining concern.

Reviewer_8wpEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GL-JEPA, a graph–language pretraining framework that predicts frozen text embeddings of paired descriptions from graph representations. A GraphGPS encoder processes directed structure and node text, while a Transformer predictor reconstructs the description embedding through normalized regression. An initial graph-context pretraining stage predicts masked-node representations from an exponential-moving-average target encoder. Downstream evaluation uses frozen representations for graph prediction and retrieval, with query-conditioned predictor adaptation for graph question answering. The experiments compare latent prediction with autoregressive generation and contrastive alignment. The reported results show better average performance on node, link, and graph prediction than the compared alignment methods. Alignment remains stronger on retrieval.

Critical strengths
  1. The paper introduces a joint-embedding predictive architecture for graph–language learning. Predicting frozen description embeddings provides language supervision without token-level reconstruction and supports transfer across multiple graph tasks.

  2. The paper provides insight into when contrastive alignment suppresses target-relevant information. They demonstrate that latent prediction can retain such information without requiring factor-specific hard negatives.

  3. Experiments cover multiple graph tasks, and the ablation study examines the main architectural and training choices.

Critical weaknesses
  1. The feature-suppression diagnostics use simplified settings and mainly assess attribute recovery. They do not establish whether improved structural information retention explains the downstream gains.

  2. For link prediction, it does not specify whether evaluated positive edges are removed before computing graph representations and positional features. Clarifying the experimental settings is necessary to assess the validity of the reported results.

  3. For question-answering tasks, it mixes adapted binary scoring with published generative results. Differences in supervision and answer formats limit comparability, while task-specific adaptation data and parameter counts need clarification.

Questions for authors
  1. Does latent prediction also improve recovery of the structural factor in the existing synthetic diagnostic? Reporting this result would better connect the analysis to graph representation learning.

  2. Clarify whether evaluated positive edges are removed before computing graph representations and positional features for link prediction. This would help rule out potential information leakage.

  3. Could the dual-positive accuracy evaluation include a baseline that averages node-text embeddings from the frozen Text Encoder and uses the same cosine scoring? This would help identify the contribution of the Graph Encoder and Predictor.

3 comments

We sincerely thank Reviewer 8wpE for the thoughtful questions. Changes made in response to this review are highlighted in red in the revised paper, and changes that also answer Reviewer XbvX are highlighted in purple.


Response to W1 and Q1: Does latent prediction also recover structural factors, and does retained structure explain the gains?

In the original diagnostic both objectives recover the structural factor at 100%, so we added a structural factor that is in the target but not needed to tell pairs apart. At width 8, easy-negative Alignment recovers it at 19.5% (chance 12.5%) and latent prediction at 100% (Section 3.3, Appendix C Table R2). On the seven real TAGs, linear probes on the Graph Encoder outputs of the Table 1 checkpoints show the same for the number of nodes and the root degree on every dataset, and the two are similar on triangles.

Probe R² (%)AlignmentGL-JEPAGL-JEPA higher
Number of nodes76.385.87/7
Root degree69.987.07/7
Triangles43.641.81/7

To link structure to accuracy, we erased the three factors with LEACE from the embedding each model scores against the class prompts and re-ran zero-shot node classification on the test ego-graphs. Alignment drops 2.89 points and GL-JEPA 0.78, so the gap grows from 1.7 to 3.8 points, while erasing permuted values changes either model by at most 0.31 (Appendix C, Table R2(d)). Alignment depends more on these statistics, and the GL-JEPA advantage holds without them.

Node accuracy (%)NoneErasedPermuted
Alignment57.1954.3056.88
GL-JEPA58.9158.1359.01

Response to W2 and Q2: Are evaluated positive edges removed before computing link representations?

They were not, and the manuscript did not say so. Appendix B now states this and reports a re-evaluation that removes every evaluated positive edge in both directions and rebuilds the views and positional features. Appendix B, Table R1 reports the per-dataset AUC for all seven datasets. GL-JEPA keeps the highest mean AUC in both comparisons, and Section 3.4 now refers to these results.

Mean link AUC over seven datasets (%)

MethodInductiveTransductive
GL-JEPA (Table 2)62.775.6
GraphCLIP57.871.6
ADAligner50.653.3
GL-JEPA (Table 1)62.375.7
Alignment59.871.9

Response to W3: How are the QA baselines adapted and scored?

In the original Table 3, GraphGPT, LLaGA, and GraphSolver were scored on two candidate answers like GL-JEPA, and TEA-GLM, GOFA, G1-3B, and GraphWiz on generated answers. In the revision, every model is adapted, so the comparison is not zero-shot, and is scored by the likelihood of two candidate answers ({yes, no}, or {support, counter} on ExplaGraphs). Learning rate, batch size, and epochs follow each model's official script where available, and GL-JEPA adapts only the Predictor and output projection with the Graph Encoder frozen. Changed values are shown in purple in Table 3, and Appendix E gives the settings and data splits (Tables R6 and R7).

Accuracy (%). LR, batch, and epochs are the GraphInstruct settings. Bold marks the best value per benchmark.

ModelTrainableSizeLRBatchEpochsGraphInstructExplaGraphsNLGraph Conn.NLGraph Cyc.
GL-JEPA754M1B1e-58579.861.086.399.5
GraphGPT40.0M (LoRA)7B2e-416575.255.149.955.0
LLaGA2.0M (projector)7B2e-316562.451.350.151.3
GraphSolver335.5M (LoRA)8B1e-448554.958.764.252.9
TEA-GLM134.8M (graph module)7B2e-52552.759.054.252.9
GOFA1,006.9M (GNN layers)7B1e-416173.357.054.252.9
G1-3B239.5M (LoRA)3B1e-448281.893.398.490.6
GraphWiz6,738.4M (full, by authors)7B–––59.178.067.452.9

Data splits and answer candidates (Table R7)

BenchmarkTrainValidationTestCandidates
GraphInstruct11,4331,2672,000{yes, no}
ExplaGraphs1,659553554{support, counter}
CLeGR-Facts12,6284,8844,488{yes, no}
NLGraph2,539281562{yes, no}

Response to Q3: How does a node-text mean baseline perform on DPA?

We added it to Figure 3(b) (red bars) and Table 10 in Appendix E, averaging the frozen Text Encoder embeddings of the node texts in each graph and using the same cosine rule. It reaches 54.2% DPA on WebNLG, below GL-JEPA (60.8%), and 87.0% on DART, above GL-JEPA (83.6%). DART negatives describe other graphs, so node texts are enough to reject them. WebNLG negatives change one fact about the same entities, and there the node-text mean rejects 32.3% of relation-changed and 16.8% of direction-changed negatives, against 57.2% and 42.2% for GL-JEPA, which reads the relations from the graph.

Dear Reviewer 8wpE,

Thank you again for your detailed questions. We would like to check whether our response resolves them before the discussion period closes. With every evaluated edge removed, GL-JEPA keeps the highest mean link AUC (Table R1), and erasing structural factors lowers node accuracy by 2.89 points for Alignment and 0.78 points for GL-JEPA (Table R2). Please let us know if any point needs further clarification.

Dear Reviewer 8wpE,

We would like to kindly follow up once more before the discussion closes. In short: with every evaluated edge removed, GL-JEPA keeps the highest mean link AUC (Table R1), and erasing structural factors lowers node accuracy by 2.89 points for Alignment versus 0.78 for GL-JEPA (Table R2).

Please let us know whether these address your questions, or if anything remains unclear.

Reviewer_83xeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript introduces a framework, called GL-JEPA, for graph-language latent prediction. Their method firstly utilizes self-supervised learning with a masked-node prediction objective to obtain an initial graph encoder for the next stage. The graph encoder is then used together with a frozen text encoder and a predictor to construct a JEPA-style framework to produce hidden states that align graph structural information with semantic description information. Several downstream tasks from different perspective and across node- link- graph-levels have been conducted and the proposed method shows competitive results against chosen baseline methods.

Critical strengths
  • S1: The downstreams tasks selected contain a wide variety of questions which demonstrate a good potential of the proposed method, though I have concerns regarding some details in their settings.
  • S2: Both graph context pretraining stage and the graph-language latent prediction stage are presented clearly. The intuition and writing of each stage are easy to follow.
Critical weaknesses
  • W1: My main concern is the novelty of the proposed method. To my best knowledge, the first stage is similar to GraphJEPA [1] and NodeJEPA [2], while the second stage is a JEPA adaptation of GraphCLIP [3].
  • W2: The claim of identifying challenges in language supervision for TAGs is supported only by empirical results within the scope of the proposed method.
  • W3: Some LM-based components are relying on relatively old models, like Llama3.2-1B and all-MiniLM-L6. Counterintuitively, the experient results in Table 4 suggest that using newer models may degrade the performance. Also, a scaling analysis is missing, particularly in whether the GL-JEPA can benefit from more pretraining datasets or more model parameters.

[1] Graph-level Representation Learning with Joint-Embedding Predictive Architectures. 2025

[2] NodeJEPA: Structure-Conditioned Latent Prediction for Node-Level Graph Self-Supervised Learning, 2026

[3] GraphCLIP: Enhancing Transferability in Graph Foundation Models for Text-Attributed Graphs. 2025

Questions for authors
  • Q1: Why does the graph context pretraining also need a JEPA-style design instead of other contrastive learning methods? For example, if the second stage directly starts with a graph encoder pretrained by GraphMAE, then perform the graph-language alignment.
  • Q2: Following the first question, is the sequential pretraining necessary? Can the objective losses from both stages be combined into one loss, like l=lstructure+λlsemanticl = l_{structure} + \lambda l_{semantic}?
  • Q3: For equation 3, can you clarify more on how the eGe_G and eNe_N are obtained and why they are added to the node/graph projected states?
6 comments

Thank you for the questions. Changes for this review are in blue in the revised paper.


Response to W1 and Q1: How do the two stages relate to Graph-JEPA, NodeJEPA, and GraphCLIP, and why not start from GraphMAE?

The mapping is correct, and Stage 1 is not our contribution. Stage 1 (GCP) only initializes the Graph Encoder. We kept it a plain JEPA that matches masked-node representations to an EMA target encoder, without the hyperbolic coordinates of Graph-JEPA or the structure-conditioned design of NodeJEPA, so that the effect of Stage 2 stays visible. We chose JEPA over GraphMAE, which reconstructs input features, so that both stages predict representations.

The contribution is Stage 2, which predicts the description embedding instead of matching to it. GraphCLIP's InfoNCE only has to tell pairs apart, so it can drop factors that the negatives do not require [1]. Regressing the frozen Text Encoder target makes the optimal prediction its conditional mean direction, so factors the graph can predict are more likely to stay. The same Predictor also takes questions, so it answers graph questions, which GraphCLIP cannot. Table R5 (Appendix D) includes your suggested GraphMAE → Alignment (last row). Latent prediction beats Alignment under both initializers, and GCP is the better initializer for latent prediction.

Initialization → Stage 2NodeLinkGraph
GCP → latent prediction (GL-JEPA)60.7075.5858.15
GraphMAE → latent prediction59.2574.8856.10
GCP → Alignment59.7372.7948.60
GraphMAE → Alignment58.4273.9852.48

Response to W2: Are the challenges supported only within the proposed method?

Both limitations are already reported outside graphs. Contrastive learning suppresses features that pair discrimination does not need [1, 2, 3], and token prediction fits the wording of the reference sentence [4]. We therefore changed "We identify two challenges" to "We address two challenges" (Section 1).

Sections 3.1 to 3.3 change only the loss, with the Graph Encoder, data, and updates fixed (Appendix E), and follow the measurements of prior work (3.1 [2, 3], 3.2 [4], 3.3 [1, 5]). Published GLMs differ in encoder and data as well, so they cannot isolate the loss. Outside our implementation, the released GraphCLIP checkpoint shows the same pattern as Alignment in Table 1, with higher Retrieval (75.55 vs. 73.41) and lower Node, Link, and Graph (Table 2).


[1] Robinson, 2021, Can Contrastive Learning Avoid Shortcut Solutions?

[2] Li, 2023, Addressing Feature Suppression in Unsupervised Visual Representations

[3] Bleeker, 2023, Reducing Predictive Feature Suppression in Resource-Constrained Contrastive Image-Caption Retrieval

[4] Chen, 2026, VL-JEPA: Joint Embedding Predictive Architecture for Vision-Language

[5] Chen, 2021, Intriguing Properties of Contrastive Losses

Response to W3: Why older language models, and how does GL-JEPA scale?

We use these models to show that the method does not depend on the latest models and to use the same Text Encoder as the released GraphCLIP (all-MiniLM-L6-v2), which makes the comparison direct. On TAGs, larger text embedding models have been reported not to give consistently better graph performance, with the pretraining objective mattering more [6]. Table 4(c) is also task-dependent, so we removed the interpretation "affects transfer beyond encoder size".

We ran two scaling studies, added to Table 4 ((g), (e)) of the revision. The best value in each column is in bold.

(a) Predictor layer scale. Eight blocks give the highest Node, Retrieval, and GQA, and sixteen blocks lower Retrieval by 11.5 and GQA by 7.8 points.

Predictor blocksNodeLinkRetrievalGQA
459.9975.4764.6979.40
8 (GL-JEPA)60.7075.5873.4179.75
1659.4675.7061.8971.95

(b) Pretraining data scale. Node, Link, and Retrieval rise from 25% to 100% of the data, and Retrieval gains the most (54.5 to 73.4).

Pretraining dataNodeLinkRetrievalGQA
25%57.0074.5154.5480.50
50%57.2874.7960.9476.40
75%57.5375.0563.7678.70
100% (GL-JEPA)60.7075.5873.4179.75

Response to Q2: Is sequential pretraining necessary?

The two losses can be combined. We trained the joint objective L_GCP + L_GL-JEPA (λ = 1) from a random Graph Encoder with the same updates, data order, and evaluation as Stage 2, and added it to Table 4(a) of the revision. It is close to the sequential pipeline on Link and GQA and 3.2 points lower on Node. We keep two stages because the targets differ. GCP regresses onto an EMA of the encoder being trained, and latent prediction regresses onto fixed description embeddings, so in a single loss the fixed target can pull the encoder before it has learned structure.

ConfigurationNodeLinkRetrievalGQA
Full pipeline (GCP → latent prediction)60.7075.5873.4179.75
Joint Pretraining (L_GCP + L_GL-JEPA)57.4975.1371.3980.30

Response to Q3: How are e_G and e_N obtained, and why are they added?

e_G and e_N are learned Predictor vectors, initialized from a truncated normal and trained with the loss in Eq. 5 (Section 2.3). Rotary positions encode only relative distances and cannot mark which token is the graph summary, so, like BERT segment embeddings, e_G is added to the summary token and e_N to every node token to mark the token type. Adding instead of concatenating keeps the input width of the Llama blocks.


[6] Chen, 2024, Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs

Dear Reviewer 83xe,

Thank you again for your review. As the discussion period is coming to an end, we would like to ask whether our response addresses your concerns. Starting from GraphMAE, latent prediction still exceeds Alignment on every task (Table R5), and Table 4 now reports the Predictor-depth and data-scale studies you asked for. We are glad to answer any remaining question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.