- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors identify two potential limitations of existing graph–language pretraining approaches. Autoregressive token prediction requires reconstructing a particular linguistic realization of a graph description, while direct graph–text alignment may suppress information that is unnecessary for distinguishing matched graph–text pairs but remains useful for downstream graph tasks. To address these issues, GL-JEPA uses a frozen Text Encoder to obtain continuous representations of paired graph descriptions. A Graph Encoder processes graph structure and node-text attributes, while a trainable Predictor estimates the corresponding description representation. Training minimizes a normalized prediction loss without autoregressive token reconstruction or contrastive negatives. In the controlled comparison, GL-JEPA outperforms direct Alignment on node classification, link prediction, and graph classification, while Alignment achieves higher graph–text retrieval performance. A nearly parameter-matched Linear Predictor variant also performs better than Alignment on the three graph prediction tasks.
Critical strengths
Strength 1
The paper addresses a meaningful problem in graph representation learning: how to use paired natural-language descriptions as supervision without requiring exact token reconstruction or relying exclusively on contrastive graph–text matching. An important strength is the controlled comparison in Table 1. GL-JEPA improves over Alignment on node classification, link prediction, and graph classification. Although the full GL-JEPA Predictor has substantially more parameters than Alignment, the authors also evaluate a nearly parameter-matched Linear Predictor. This smaller variant retains advantages over Alignment on all three graph prediction tasks, suggesting that the improvements are not explained solely by the larger Transformer Predictor.
Strength 2
The complete-system frozen-transfer results show that GL-JEPA performs competitively across heterogeneous benchmarks, achieving the highest mean scores among the evaluated methods on node, link, and graph classification. In the synthetic experiment, Alignment with easy negatives achieves approximately 12.7% probe accuracy on the target-relevant factor, while GL-JEPA reaches 95.6%. The WikiBio diagnostic shows a similar qualitative pattern. The combination of controlled diagnostics, frozen-transfer benchmarks, and component ablations is a substantial strength of the paper.
Critical weaknesses
Weakness 1
A central motivation of the paper is that direct graph–text alignment may suppress information that is unnecessary for distinguishing matched from mismatched pairs but remains useful for downstream tasks. However, the results demonstrate that the extent of feature suppression depends strongly on the construction of contrastive negatives. In the synthetic experiment, easy-negative Alignment achieves 12.7% probe accuracy on factor B, compared with 95.6% for GL-JEPA. However, hard-negative Alignment achieves 99.9%, exceeding GL-JEPA. A similar pattern appears in the WikiBio experiment, where hard-negative Alignment achieves 100% occupation probe accuracy, compared with 77.1% for GL-JEPA. The results suggest that feature suppression is not an inherent limitation of graph–text alignment, but can arise when the contrastive objective is satisfied using shortcuts provided by the negative-sampling strategy. The current evidence supports a narrower conclusion: predicting the full description target can preserve additional factors without requiring task-specific hard-negative construction under the examined settings. It remains unclear whether the same degree of feature suppression occurs in realistic graph–language pretraining with larger representations and more diverse negative sampling.
Weakness 2
The graph question-answering results in Table 3 are presented as evidence that GL-JEPA learns graph representations that support competitive graph reasoning. But, the evaluation compares GL-JEPA with systems that do not necessarily use the same answer-prediction protocol. As described in Section 3.5, GL-JEPA performs discriminative candidate-answer scoring using a query-conditioned Predictor and a frozen answer Text Encoder. In contrast, several of the compared methods use generative language-model-based answering, potentially involving different output constraints, decoding procedures, and answer formats. This distinction is very important because Table 3 reports very large advantages on some benchmarks, including 99.48% accuracy on NLGraph Cycle. Without harmonizing the output protocols, it is difficult to determine how much of these improvements comes from the quality of the learned graph representations rather than differences in the answer interface. Furthermore, although the Graph Encoder remains frozen, the query-conditioned Predictor and output projection are adapted using task-specific supervision. The results therefore demonstrate the usefulness of the frozen graph representations under a trainable readout, rather than zero-shot reasoning by an entirely frozen model.
Questions for authors
Q1
The feature-suppression experiments show that hard-negative Alignment substantially improves factor recovery and even outperforms GL-JEPA in both diagnostic settings. Could the authors clarify what advantage latent prediction retains when contrastive alignment uses sufficiently informative negatives?
Q2
Section 3.5 and Appendix E describe GL-JEPA as a discriminative model that scores candidate answers through a query-conditioned Predictor rather than generating free-form text. Could the authors clarify whether the baselines in Table 3 use equivalent candidate-answer constraints and evaluation procedures? How would the comparison change if all methods were evaluated using the same binary answer-scoring protocol?
Q3
Section 2.3 states that the Predictor assigns consecutive rotary positions to graph-derived node tokens. Since node ordering is generally arbitrary in graph representations, does the resulting Predictor preserve permutation invariance? Could the authors report whether permuting node IDs while preserving node features and graph connectivity changes the predicted description representation or downstream performance?
Q4
The method relies on a frozen Text Encoder to provide continuous supervision for graph representations. But, the target embedding may not preserve every semantic or structural detail contained in the original description. What evidence supports the assumption that predicting the full text embedding preserves a broader range of graph-relevant information than direct alignment?
5 comments
Response to W2 and Q2: What changes under a common answer scoring?
In the original Table 3, three baselines were scored on two candidate answers like GL-JEPA, and the other four on generated answers. In the revision, every model is adapted, so the comparison is not zero-shot, and is scored by the likelihood of two candidate answers ({yes, no}, or {support, counter} on ExplaGraphs). Training settings follow each model's official script where available, and GL-JEPA adapts only the Predictor and output projection with the Graph Encoder frozen. Changed values are shown in purple in Table 3, and Appendix E gives the settings and data splits (Tables R6 and R7).
Accuracy (%). Bold marks the best value per benchmark. On NLGraph, 52.9 (Cyc.) and 54.2 (Conn.) come from answering yes to every question (Appendix E).
| Model | Trainable | Size | GraphInstruct | ExplaGraphs | NLGraph Conn. | NLGraph Cyc. |
|---|---|---|---|---|---|---|
| GL-JEPA | 754M | 1B | 79.8 | 61.0 | 86.3 | 99.5 |
| GraphGPT | 40.0M (LoRA) | 7B | 75.2 | 55.1 | 49.9 | 55.0 |
| LLaGA | 2.0M (projector) | 7B | 62.4 | 51.3 | 50.1 | 51.3 |
| GraphSolver | 335.5M (LoRA) | 8B | 54.9 | 58.7 | 64.2 | 52.9 |
| TEA-GLM | 134.8M (graph module) | 7B | 52.7 | 59.0 | 54.2 | 52.9 |
| GOFA | 1,006.9M (GNN layers) | 7B | 73.3 | 57.0 | 54.2 | 52.9 |
| G1-3B | 239.5M (LoRA) | 3B | 81.8 | 93.3 | 98.4 | 90.6 |
| GraphWiz | 6,738.4M (full, by authors) | 7B | 59.1 | 78.0 | 67.4 | 52.9 |
Response to Q3: Does permuting node IDs change the results?
The Predictor is not exactly permutation invariant, but node order has a small effect. Structure enters through the order-independent Graph Encoder, and order enters only through the rotary positions of node tokens, which leave node states unchanged. Permuting node IDs five times on Cora and CiteSeer with the Table 1 checkpoint keeps the mean cosine between predicted descriptions at 0.997 or higher and moves zero-shot node accuracy by at most 0.23 points on average, one or two test nodes (Table R4).
Response to Q4: What supports that predicting the frozen target preserves more than alignment?
Both objectives use the same frozen target, so information missing from the target is missing for both. Our claim concerns factors that are in the target but not needed for matching. Regression must fit every target coordinate, while a contrastive loss gives little supervision to factors that do not separate positives from negatives. In WikiBio, the target encodes occupation at 100%. Easy-negative Alignment lowers it to 27.1% (chance 25.0%), below the 64.6% of the untrained graph state, and latent prediction raises it to 77.1% (Figure 4, Table R3). Structural factors behave the same way (Table R2). The Limitations section discusses the limits of the target.
Dear Reviewer XbvX,
Thank you again for your questions. Before the discussion period ends, we would appreciate knowing whether our response answers them. In the WikiBio sweep, Alignment loses nationality as more negatives share it, while latent prediction keeps both factors without tuning (Table R3), and all QA baselines are now scored on two candidate answers (Table 3). We would be glad to discuss any remaining concern.
Thank you for the questions. Changes for this review are in green in the revised paper, and changes shared with Reviewer 8wpE are in purple.
Response to W1 and Q1: What does latent prediction offer when negatives are informative?
Latent prediction has no negatives to design, so no factor must be chosen for protection in advance. A contrastive loss keeps what separates positives from negatives and can suppress the rest [1, 2, 3], so harder negatives recover some features and suppress others [4]. In WikiBio, hard-negative Alignment reaches 100% on occupation but is at chance on nationality, which its candidates share (Figure 4, Table R3). With uniform negatives (β = 0), both objectives keep both factors, with Alignment 2.1 points higher on each. As more negatives share the anchor's nationality, Alignment loses nationality (35.4% at β = 2, chance from β = 4), while latent prediction keeps both without tuning. Section 3.3 states this narrower conclusion.
In the structural diagnostic, Alignment with random negatives or one hard negative per anchor leaves structure near chance up to width 8 (19.5% and 20.6%, chance 12.5%) and recovers it only with structure-defined positives, while latent prediction recovers it at every width (Table R2). Latent prediction can lose a factor when the graph state is small. The 128-way A (chance 0.8%) stays near chance up to width 8 and returns at width 32 (96.7%). In the Table 1 checkpoints (full corpus, 384-dimensional Graph Encoder), GL-JEPA graph states predict the number of nodes and the root degree better than Alignment on all seven datasets (Table R2(c)), and GL-JEPA is higher on three of the four transfer tasks (Table 1).
[1] Chen, 2021, Intriguing Properties of Contrastive Losses
[2] Li, 2023, Addressing Feature Suppression in Unsupervised Visual Representations
[3] Bleeker, 2023, Reducing Predictive Feature Suppression in Resource-Constrained Contrastive Image-Caption Retrieval
[4] Robinson, 2021, Can Contrastive Learning Avoid Shortcut Solutions?