acceptodds
Under review as a conference paper at ICLR 2027

LeGramJEPA: Negative-Free Multimodal Graph Foundation Models in Latent Space with Volumetric Alignment and SIGReg

Abstract

Self-supervised learning on text-attributed graphs enables graph foundation models that transfer to out-of-distribution graphs without labels. The standard approach aligns a graph encoder and a text encoder with a contrastive objective, pulling positive graph-text pairs together and pushing mismatched pairs apart. This depends on large numbers of negative pairs, and extending it beyond two modalities requires a pairwise alignment term for each pair of modalities, so the number of terms grows quadratically. We present LeGramJEPA, a latent-space approach to multimodal graph self-supervised learning that uses no negatives and whose alignment cost grows linearly in the number of modalities. LeGramJEPA replaces the contrastive term with a volumetric alignment objective and constrains each encoder’s output distribution toward an isotropic Gaussian with SIGReg to prevent embedding collapse. The volume term couples all modalities in a single quantity per anchor, reducing the number of alignment terms from quadratic to linear in modalities compared to cross-modal methods. Pretraining with LeGramJEPA is competitive with contrastive baselines on zero-shot node classification and improves over them on link prediction. Under a three-modality (graph, text, image) training and transfer scheme, LeGramJEPA consistently outperforms its contrastive counterpart across modality pairs

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph Learning, MultiModal, SIGReg, JEPA, Information Maximization, Text-Attributed Graphs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8u4gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies negative-free graph-text and graph-text-image representation learning for transfer across text-attributed graphs. It proposes LeGramJEPA, which replaces pairwise cross-modal prediction with a per-anchor Gram-determinant volume loss and uses SIGReg to constrain each modality’s embedding distribution. The method is evaluated on zero-shot node classification and link prediction, plus linear probing in a three-modality setting.

Critical strengths
  1. The paper provides a clear computational formulation with a meaningful efficiency advantage.

  2. The empirical evaluation demonstrates that the negative-free objective is viable in several transfer settings.

  3. The figures are clear and helpful to understand the framework.

Critical weaknesses
  1. Minimizing the Gram determinant only enforces linear dependence, not full alignment across modalities.

  2. The three-modality experiments rely primarily on downstream linear probing, without directly measuring whether all modalities are jointly aligned. Cross-modal retrieval experiments or embedding alignment diagnostics would better substantiate the proposed objective.

  3. Appendix D.1 reports the best-performing scoring direction separately for each target dataset, potentially using target test labels for model selection. This undermines the strict zero-shot setting and may introduce an unfair advantage over GraphCLIP.

  4. LeGramJEPA underperforms GraphCLIP on four of seven zero-shot node classification datasets. Moreover, the three-modality model fails to outperform the best two-modality variant, raising concerns about the effectiveness of the proposed multimodal extension.

  5. The efficiency claims are supported primarily by loss-only benchmarks on synthetic embeddings rather than end-to-end pretraining experiments. Furthermore, the proposed objective is slower than InfoNCE for batch sizes below approximately 2,554, making its practical efficiency advantages unclear.

Questions for authors
  1. How does the implementation or optimization dynamics rule out rank-deficient but non-aligned multimodal solutions of the form z3=z1z_3 = z_1 while z2z_2 remains independent, for which every three-modality determinant in Equation (6) is zero?
1 comment

We thank the reviewer for the careful reading. We address the weaknesses below.

W2. No direct measure of three modality alignment.

MM-Graph evaluates Ele-Fashion and Goodreads-NC only with supervised GNNs trained on each target. A frozen-encoder probe on the same splits allows us to compare the pretrained model next to the supervised results (Table 3; App. D.2). We believe that the node classification and link prediction results already show that alignment is learned via the volumetric term.

W3. Scoring direction chosen per target dataset.

Thank you for this question.We will clarify App. D.1. A single scoring direction, [cos(z_g, h_t(z_t))], is used for all seven target datasets; it is not chosen per dataset. We will add further clarification in App. D.1 to make the zero-shot scoring clear.

W4. Performance relative to GraphCLIP, and three modalities.

The focus of this paper is to demonstrate that a negative-free self-supervised objective is competitive with the contrastive methods in graph learning. Under three modalities, the focus here is to demonstrate that a model trained with simpler k volumetric alignment terms instead of k(k-1) still outperforms both GraphCLIP across each modality pair.

W5. Efficiency Claims

Thank you for raising this point, the benchmark in App. H mainly focuses on the loss with synthetic embeddings, therefore following your suggestions we have included an extra experiment, instead we now include the timed forward pass and peak memory inside a full pretraining run with the encoders. We compare against GraphCLIP's loss, symmetric InfoNCE, and the cross pairwise objective (Eq. 15).

kGlobal BVolume (ms / MiB)Pairwise L_cross (ms / MiB)InfoNCE, all pairs (ms / MiB)GraphCLIP (ms / MiB)
29001.86 / 183161.36 / 183180.77 / 180270.95 / 17796
236002.08 / 677911.67 / 677921.86 / 675961.52/ 67596
272008.72 / 677918.43 / 6779010.15 / 6759518.86 / 67268
21440015.10 / 6779113.92 / 6779025.92 / 6855152.59 / 68287
3720011.48 / 6804610.87 / 6804819.33 / 67753n/a
31440019.29 / 6804518.57 / 68048OOMn/a
  • Two modalities. At global B = 7,200, our loss takes 8.72 ms, InfoNCE takes 10.15 ms and GraphCLIP takes 18.86 ms. At B = 14,400 ours grows 1.7× to 15.10 ms, while InfoNCE grows 2.6× to 25.92 ms and GraphCLIP 2.8× to 52.59 ms.
  • Three modalities. At B = 7,200, ours takes 11.48 ms and InfoNCE summed over the three pairs takes 19.33 ms. At B = 14,400 ours takes 19.29 ms, and the InfoNCE OOM.
Reviewer_UkczOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies self-supervised pretraining and cross-graph transfer on text-attributed graphs (TAGs). The authors propose LeGramJEPA, a multimodal pretraining method that does not need negative samples. It uses a predictor head, stop-gradient, and a Gram-determinant volume objective to align representations across modalities. To prevent collapse without negatives, it applies SIGReg to each modality so the embeddings follow an isotropic Gaussian distribution. For k modalities, this replaces the k(k-1) directed pairwise alignment losses with k volume terms. The method is evaluated on zero-shot node classification, zero-shot link prediction, and multimodal linear probing. The authors also test how the time and memory of the loss calculation scale with batch size.

Critical strengths
  • In Section 4, each anchor modality uses a volume term to align with all other modalities at once, while SIGReg avoids collapse. Combining volume alignment with negative-free graph learning is a solid idea, as it offers an alternative to InfoNCE in graph-text pretraining and reduces the number of alignment terms from k(k-1) to k.

  • The method gets good transfer results without negative samples. In Table 1, LeGramJEPA is competitive with GraphCLIP on zero-shot node classification (better on some graphs, worse on others). The gains are clearer on zero-shot link prediction (Table 2, Figure 3) and 3-modality linear probing (Table 3), showing that it learns useful node embeddings without relying on negative pairs.

Critical weaknesses
  • Reducing the number of alignment terms from k(k-1) to k (Section 4.2) does not clearly translate into faster real-world training. Each volume term still has to build a k x k Gram matrix and compute its determinant, which also depends on k. More importantly, the efficiency test in Section 5.5 and Appendices G-H fixes k=2, varies the batch size B, and only runs on random tensors without encoders. This only shows linear scaling in B for the loss alone; it does not show how the cost scales with k, or how much time/memory is saved when the graph, text, and image encoders are included. Also, for B < 2554 (which covers normal batch sizes), this loss is slower than InfoNCE. So it is unclear if the method actually saves training time or memory in the 2- and 3-modality setups used in the paper.

  • There is no ablation isolating the volume objective from pairwise prediction. Section 4.3 uses the same encoders as GraphCLIP, but switching from GraphCLIP to LeGramJEPA changes the alignment loss, the predictor/stop-gradient setup, and the SIGReg regularizer all at once. Section 4.1 (Eq. 3) and Appendix E (Eq. 15) explicitly define the pairwise prediction loss L_cross (from LeVLJEPA), yet there are no results comparing the volume loss against L_cross while keeping the encoders, predictors, and SIGReg fixed. Without this baseline, it is hard to tell how much of the gain comes from the volume term itself versus the negative-free SIGReg framework.

Questions for authors
  1. Could you separate the complexity in the number of modalities k from the complexity in the batch size B? For your actual 2- and 3-modality setups (with encoders included and at your actual training batch sizes), what are the step times and peak memory compared to GraphCLIP and pairwise L_cross?

  2. If you keep the encoders, predictors, stop-gradient, and SIGReg identical, how does the volume objective compare to the pairwise prediction loss (L_cross in Section 4.1 / Appendix E) in terms of downstream accuracy and training cost?

2 comments

We thank the reviewer for the careful reading and suggestions for experiments. We address the weaknesses below.

W1/Q1. Cost in K and B, with encoders.

We would like to clarify our claims. The volume loss is quadratic in k in compute (one k×k Gram per anchor), the same order as pairwise L_cross (Eq. 15). What is linear in k is the number of alignment terms (k vs k(k−1)). The advantage over contrastive alignment comes from removing the B×B negatives. We will reword the abstract and contributions accordingly.

However, despite this we include an end-to-end pretraining efficiency study varying batch size and k.

kGlobal BVolume (ms / MiB)Pairwise L_cross (ms / MiB)InfoNCE, all pairs (ms / MiB)GraphCLIP (ms / MiB)
29001.86 / 183161.36 / 183180.77 / 180270.95 / 17796
236002.08 / 677911.67 / 677921.86 / 675961.52/ 67596
272008.72 / 677918.43 / 6779010.15 / 6759518.86 / 67268
21440015.10 / 6779113.92 / 6779025.92 / 6855152.59 / 68287
3720011.48 / 6804610.87 / 6804819.33 / 67753n/a
31440019.29 / 6804518.57 / 68048OOMn/a

At our single-node batch size the volume term is slower than InfoNCE. As B grows, InfoNCE and GraphCLIP grow faster: from B = 7200 to 14400, volume rises 1.7×, InfoNCE 2.6× and GraphCLIP 2.8×. At k= 3 at B = 7200 raises volume by 1.3× and InfoNCE by 1.9×. At k = 3, B = 14400, InfoNCE OOM. Volume and pairwise L_cross cost about the same throughout.

W2/Q2. Isolating the Volume Term

We include an ablation swapping the alignment term L_cross (Eq. 3/15) vs volume (Eq. 6), with encoders, predictors, stop-gradient and SIGReg held fixed.

Zero-shot node classification (accuracy, %):

MethodParamsCoraCiteSeerWikiCSInstagramEle-PhotoEle-Comp.Books-Hist.
GraphCLIP150M67.31 ± 1.7663.13 ± 1.1370.19 ± 0.1064.05 ± 0.3453.40 ± 0.6462.04 ± 0.2153.88 ± 0.35
GraphCLIP w/o Freeze150M61.29 ± 1.3761.66 ± 1.0158.92 ± 0.1060.15 ± 0.7545.90 ± 0.6553.24 ± 0.2629.40 ± 0.31
Pairwise L_cross + SIGReg150M63.10 ± 0.9061.29 ± 2.2756.56 ± 1.0564.11 ± 1.2558.21 ± 0.3055.38 ± 0.2938.42 ± 0.`35
LeGramJEPA (volume + SIGReg)150M65.68 ± 1.3667.40 ± 2.4662.32 ± 0.9265.04 ± 0.8954.86 ± 0.6461.14 ± 0.3948.47 ± 0.78

Zero-shot link prediction (AUC, %):

MethodCoraWikiCSBooks-Hist.
GraphCLIP83.15 ± 0.7692.67 ± 0.1096.04 ± 0.05
Pairwise L_cross + SIGReg89.38 ± 0.3878.75 ± 0.0998.06 ± 0.03
LeGramJEPA (volume + SIGReg)97.77 ± 0.3293.90 ± 0.0398.57 ± 0.02

The volume term beats pairwise L_cross on 6 of 7 targets for node classification and on all 3 for link prediction, at nearly the same training cost (W1 table). There is a clear advantage from the volume term itself rather than from the SIGReg framework. In some cases, the pairwise L_cross + SIGReg, hurts performance as in WikiCS. This suggests that volume alignment is better than a cross-modal one. We will add both tables to Sec. 5.

Thank you for the response and the additional experiments.

A few issues still remain. The ablation against L_cross is only run for k=2, without a comparison in the 3-modality setting, which is the main motivation for volumetric alignment. Also, in Table 3, the 3-modality model (G-T-I) still does not outperform the 2-modality one (G-T). For efficiency, the end-to-end speed advantage only shows up at large batch sizes (B >= 7200), while it is slower at normal batch sizes. Finally, another reviewer pointed out that for k >= 3, the determinant can be zero even when the modalities are not fully aligned, and this has not been addressed yet.

Based on these points, I will keep my score at 2.

Reviewer_aAEMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper adopt the idea of LeJEPA (a SSL method that prevents representation collapse by design and at the same time does not rely on positive & negative pairs commonly used in contrastive learning) on TAGs to demonstrate zero-shot transfer on TAGs of unseen domains. The authors also extend this idea to multi-modal graphs where each node is associated with a text feature and an image feature. All experiments, except for those on efficiency claims (Figure 4) support the novel contribution of the authors i.e., LeJEPA formulation enables building a Graph Foundation Model on TAGs. One interesting aspect is that the authors use the formulation of Gramian volume of the k-dimensional parallelotope spanned by the modality vectors from (Cicchetti et. al) for their experiments on multi-modal graphs.

Critical strengths
  1. The assumptions behind using GRAM and LeJEPA are sound, and with the right choice of the training data, LeGramJEPA performs competitively on zero-shot transfer.
  2. The choice of the experiments and baselines, and the literature review is thorough.
  3. The paper is very easy to read and understandable.
Critical weaknesses
  1. The novelty of this paper is very thin. It combines 2 existing ideas i.e. LeJEPA and GRAM (Cicchetti et. al., ICLR 2025) and implement it on multi-modal graph data. The authors do not clarify why it is challengint to combine the above 2 ideas for multi-modal graph data. For example one can use the exact same method for image+text+audio or video+image+text.
  2. The authors claim in the abstract that the alignment cost of LeGramJEPA grows linearly with number of modalities as compared to mutually pairwise contrastive learning. This is not true, since LeGramJEPA requires computing the volume of kk-dimensional parallelopiped where kk is the number of modalities, which has a quadratic computational complexity in kk. See Equation (5).
  3. In the Preliminaries, the authors write that both the graph encoder and the text encoder map the nodes to the same space Rd\mathbb{R}^d. It would have been nicer if the authors made this more general, which in turn would have enable interesting experiments like aligniing GraphGPS with any other text encoder like Qwen-Embedding or Llama.
  4. The experiment about computational efficiency in Figure 4 is unfair towards InfoNCE. For a batch size BB pairs, InfoNCE depends on the negatives B(B−1)B (B-1) cross-pairs while LeGramJEPA does not require any negatives. To test the quadractic dependency, the appropriate experiment should vary kk i.e., the number of modalities and not batch-size to show that mutually pairwise InfoNCE scales quadratically in number of modalities while LeGramJEPA scales linearly (which is also not true as raised in Weakness 2).
Questions for authors

Refer to the weaknesses.

1 comment

We thank the reviewer for the careful reading and constructive weaknesses. We address some below.

W2. Linear in the number of modalities

Thank you for noticing this point we will update the wording of the manuascript; the reviewer is correct about compute. The abstract stated “... and whose alignment cost grows linear in the number of modalities” this was a typo and should read “... and whose alignment terms grows linear in the number of modalities”. Per sample, the k anchor Gram matrices share all pairwise inner products, so forming them costs Θ(k²d). The k determinants cost Θ(k³) each, so Θ(k⁴) in total. Per batch, the volume term costs Θ(B(k²d + k⁴)), which is Θ(Bk²d) when d ≥ k² (d = 768 here). The number of alignment terms is k instead of k(k−1) (App. E), and that is the only quantity that is linear in k. Mutually pairwise InfoNCE needs k(k−1)/2 B×B score matrices, which costs Θ(k²B²d) (App. G.1, Prop. 1). So yes, both volume and InfoNCE are quadratic in k. The important difference is in B, because the volume term needs no negatives. We will change the abstract, Sec. 1.1 and the contributions to say “one alignment term per modality, with cost linear in batch size”, and we update the k-dependence in App. G.1 so this is clearer in the revised version.

W3. Shared space R^d

Each encoder keeps its own native dimension, so any text encoder can be used. The three-modality experiments already use T5 in place of MiniLM (Sec. 5). We used GraphGPS + MiniLM in the two-modality setting to match GraphCLIP (Zhu et al., 2025b). That way, differences come from the loss function and not from the encoders (Sec. 5, Architecture control). We will make the notation general (fθ: G → R^{d_g}, gφ: T → R^{d_t}, each followed by a projection to R^d).

W4. Fairness of Figure 4.

InfoNCE’s B² cost comes from its in-batch negatives. Removing negatives is the intention of our method, so the B-sweep measures the cost of that design choice. It is the relevant comparison against GraphCLIP. We agree it does not test the k-dependence, and as noted in W2, the linear-in-k claim does not hold for compute, but rather for the number of alignment terms. InfoNCE can be extended past two modalities, but this still requires a B×B score matrix per term. One option sums over all the k(k−1) terms.

To address both points, we extended the efficiency study to full end-to-end pretraining, varying both B and k, and recorded step time peak memory (MB) for each objective under the same encoders.

kGlobal BVolume (ms / MiB)Pairwise L_cross (ms / MiB)InfoNCE, all pairs (ms / MiB)GraphCLIP (ms / MiB)
29001.86 / 183161.36 / 183180.77 / 180270.95 / 17796
236002.08 / 677911.67 / 677921.86 / 675961.52 / 67596
272008.72 / 677918.43 / 6779010.15 / 6759518.86 / 67268
21440015.10 / 6779113.92 / 6779025.92 / 6855152.59 / 68287
3720011.48 / 6804610.87 / 6804819.33 / 67753n/a
31440019.29 / 6804518.57 / 68048OOMn/a

At small batch sizes, InfoNCE is faster. Above B = 3600 the order reverses. At B = 14400, the volumetric objective takes 15.10ms against 25.92ms for InfoNCE and 52.59ms for GraphCLIP. At k = 3 and B = 7200 volumetric objective step time increase by 1.32× for the volumetric objective and 1.90× for InfoNCE. At B = 14400, InfoNCE runs out of memory with three modalities while the volumetric objective continues to train.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.