acceptodds
Under review as a conference paper at ICLR 2027

Would an Untrained Encoder Do as Well? The Missing Baseline in Graph Self-Supervised Learning

Abstract

Graph self-supervised learning is judged by freezing a pretrained encoder, fitting a linear probe, and comparing accuracies on small benchmarks, where methods are separated by a point or two. These comparisons almost never ask whether the same encoder, never trained, would do as well. We run this control for graph joint-embedding predictive architectures (JEPAs) on seven benchmarks, with three partitioners, two read-outs and two implementations, including the official Graph-JEPA code, whose published accuracies we reproduce. We find that pretraining adds no detectable accuracy over a randomly initialised copy of the encoder except on the smallest dataset, and that under the published read-out the untrained encoder is significantly better on three datasets. Pooled over benchmarks, every setting excludes a common gain above one and a half points. Pretraining does reshape the representation, without improving what a linear probe reads, and the most robust partition effect, structural-entropy over METIS partitions on NCI1, is nearly as large for the never-trained encoder as for the trained one. The benchmarks set the boundary: under a corrected test they cannot resolve gains below two to six points, several times the margins that rank published methods. An untrained encoder costs one run with zero epochs, and graph self-supervised learning should be measured against it.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
self-supervised learning, graph representation learning, joint-embedding predictive architecture, structural entropy, untrained baselines, evaluation methodology, repeated cross-validation, graph classification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gYp2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper compares pretrained and randomly initialized graph JEPA encoders on small TU datasets to evaluate pretraining gains in linear-probe classification accuracy. Pretraining changes the representations, but significant accuracy gains are detected only in a few settings, and some partitioning advantages exist before training. Through official code reproduction, controlled experiments, and corrected statistical tests, the paper separates pretraining gains from architectural contributions and quantifies the benchmarks’ ability to detect small differences in accuracy.

Critical strengths
  1. The paper is clearly structured and easy to follow.
  2. Comparing with the untrained encoder helps to distinguish architectural contributions from pretraining gains. The research question is relevant to the evaluation of graph self-supervised learning.
  3. The experimental design is well-structured, covering multiple datasets, partitions, and readout configurations, providing different support for the problem. It also accounts for dependencies between cross-validation folds, reporting uncertainty and detection capability.
Critical weaknesses
  1. The paper reports dataset-level averages without analyzing which types of samples benefit from or are adversely affected by pretraining. It is unclear whether small overall differences mask opposing effects across samples.
  2. The paper does not analyze the relationship between graph size and partition granularity. It remains untested whether partitioning small graphs produces overly small or highly overlapping subgraphs that provide insufficient useful context for pretraining.
  3. The paper does not examine how the pretraining prediction objective relates to the downstream probe objective or whether the learned representations suit classification. It can also compare the joint fine-tuning of the encoder and the classifier, and compare the gradient directions and magnitudes of the two objectives at the same encoder state.
  4. Despite the partition controls and decomposition of performance differences, the paper does not sufficiently explain why random encoders perform strongly or why pretraining yields limited gains.
Questions for authors

See the weakness.

2 comments
AuthorsOpenReview ↗

Patch size against graph size, never-trained partition controls, and what the records cannot show

Thank you for the constructive review. The revised PDF marks every change in blue.

W4. Why untrained encoders are strong. Section 3.3 gives our reading. Random-walk features pooled within a partition, passed through randomly weighted GINE layers, already give a linear probe strong structural statistics. If that is right, partition effects should be visible before any training. After submission we tested this under an exploratory protocol written before the runs. Section 3.3 (page 7) and Tables 16 and 17 report every contrast it registered, including those it did not support.

  • SE fixed-K: membership. METIS given SE fixed-K's exact part sizes gains nothing detectable on NCI1 (−0.16). SE fixed-K beats that size-matched METIS by +2.03 (t_c 3.18) with never-trained encoders.
  • SE levels: consistent with the node rung. The SE ladder ends in a single-node rung, which SE fixed-K does not have. A random partition with the ladder's shape beats METIS by +1.37 (t_c 2.46), but METIS with a node rung by only +0.33, a contrast we added afterwards. The node rung's own gain (+1.04, t_c 1.91) is not significant. The registered membership test for SE levels, its lead over the random partition, is not met either never-trained (+0.62, t_c 1.45) or trained (+0.68, t_c 0.87). Its lead over METIS with its part sizes is +0.69 (t_c 1.57) never-trained and +1.32 (t_c 2.011, at the threshold) trained.
  • Part sizes add nothing detectable on NCI1 beyond the node rung, and on MUTAG no contrast is significant.

So the partition effects on NCI1 are present before training, which supports the architectural reading, though not every step of the attribution is significant. Neither the uniformity of part sizes nor the share of ring bonds cut predicted the gap across eight datasets; all four registered rank correlations were negative. What in the partition helps remains open.

Why pretraining adds so little, we cannot fully explain. One candidate is that the prediction task of our backbone is easy, because each normalised target scalar is confined to a small range (Section 2). But the official code, which has no such bound, gives the same verdict.

W2. Graph size and partition granularity. We measured patch sizes directly from the partitions, with no training (Table 18). The METIS ladder uses cluster counts up to 12 on every dataset except IMDB-M, where they run from 2 to 6. The finest rung averages 1.5 nodes per part on MUTAG, 1.6 on IMDB-B, 2.2 on IMDB-M, 2.5 on NCI1, 2.8 on ENZYMES, 3.4 on PROTEINS and 24.7 on DD (DD measured on every tenth graph). Each part is then expanded by one hop.

So you are right that the finest patches on small graphs are nearly single nodes. Larger parts do not go with larger pretraining gains, though. DD has by far the largest parts, and with METIS partitions under our read-out it has the most negative difference (−1.41). Over the seven datasets, the rank correlation between nodes per part and the METIS gain is −0.54. That is seven points and exploratory, so it argues against small patches as the main cause without ruling them out. We did not measure patch overlap. Section 5.4 (page 10) now says both.

W1. Per-sample effects. We agree this would be informative. Per-graph predictions were not stored. They could be regenerated from the saved fold models on the five datasets whose models reload exactly (not PROTEINS or IMDB-M), but we could not complete this within the discussion period. The fold records show only that the sign varies across seeds: with METIS partitions the trained encoder is ahead in 4 of 5 seeds on NCI1 and in 0 of 5 on DD.

W3. The pretraining objective and the probe. We did not examine this; joint fine-tuning and a comparison of the two objectives' gradients were not run. Section 3.2 shows only that pretraining moves the representation further from its initialisation than two initialisations are from each other, without improving what the probe reads. That exploratory result is consistent with the objective changing the features in directions a linear probe does not use, but it does not show it. Section 5.4 now names joint fine-tuning and the gradient comparison as the next tests.

Prompted by the other reviews, the revision also adds:

  • a random-effects pooling with prediction intervals (Table 14);
  • a one-sided Holm correction, which neither MUTAG win survives;
  • a plain statement that every probe in the paper uses unscaled features at a fixed strength (Sections 3.1 and 5.4);
  • a narrower statement of scope in the abstract and introduction.

I thank the reviewer for their responses. I maintain my original (generally positive) assessment of the paper.

Reviewer_BDfZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper runs the untrained-encoder control for graph JEEPAs (Graph-JEPA, HP-JEPA): on seven TU benchmarks, with three partitioners, two read-outs, and two implementations, it compares frozen pretrained encoders against randomly initialised copies of the same architecture under linear probing. Pretraining adds no detectable accuracy except on the smallest dataset (MUTAG); under the published read-out the untrained encoder is significantly better on three datasets. Pooled inverse-variance estimates exclude a common pretraining gain above ~1.4 points in every setting. A power analysis shows the benchmarks cannot resolve gains below 2.0–6.4 points, while recent comparison tables rank methods by median margins of ~0.45 points. The official Graph-JEPA code is reproduced within 0.34 points, and all fold-level records and analysis scripts are released.

Critical strengths

Exceptionally careful measurement methodology: pre-registered analysis plans with dated notes, Nadeau–Bengio-corrected significance tests, a power analysis that quantifies what the benchmarks can and cannot resolve, exact reproduction of the official code, and full release of fold-level records and table-generating scripts. This is a model of evaluation hygiene.

The control itself, once run at this level of rigor, is actionable: an untrained encoder costs one run at zero epochs, and the paper demonstrates concretely that published rankings on these benchmarks sit well inside their own noise floor. The recommendation to report this baseline is well substantiated.

Critical weaknesses

Limited novelty. That untrained/randomly initialised graph encoders rival graph SSL methods on TU benchmarks is established (Suresh et al. 2021; Trivedi et al. 2022; Khanna et al. 2025 — all cited), including explicit recommendations to report this baseline, and prior findings that several SSL methods do not beat untrained encoders. The contribution reduces to applying a known control to one further method family (JEEPAs) under a stronger statistical apparatus.

The null is confined to a protocol already known to be uninformative. Shchur et al. (2018), Errica et al. (2020), and Dwivedi et al. (2023) — all cited — already showed these benchmarks cannot separate methods; the paper's own power analysis re-derives this. By the authors' own reading a null here is a bound, not a finding about JEEPAs: up to +6.9 points of headroom remains on MUTAG, and the framing "pretraining adds no detectable accuracy" is as much a statement about the test's resolving power as about the methods.

Questions for authors

(Clarification) The AI-use statement discloses that a generative assistant contributed to conceptual framing, methodology design, implementation, running experiments, computing statistics, and drafting every section. The analysis plans and an adversarial review are described as mitigations — please clarify which experimental results and numerical claims were independently verified or re-derived by the human authors.

Given the established prior work, what is the substantive delta beyond confirming a known phenomenon for one additional method family with better statistics?

Would the null persist under a protocol the benchmarks can actually resolve (larger datasets, harder transfer splits, or fine-tuning rather than linear probing)? This determines whether the conclusion is about JEEPAs or about TU benchmarks.

1 comment
AuthorsOpenReview ↗

What is new beyond prior untrained baselines, what the bound covers, and how the numbers were checked

Thank you for the careful review. The revised PDF marks every change in blue.

Q2 / W1. The delta beyond prior untrained baselines. We agree that the control is known and has been recommended before; Sections 1 and 5.3 say so. On the TU benchmarks, the studies we cite test contrastive and non-contrastive encoders. Predictive objectives were tested at node level, and for molecular JEPA-style encoders pretrained on large external corpora, which do beat their untrained copies. The papers that propose patch-based graph JEPAs for the TU benchmarks report no untrained arm. Beyond adding one method family, the paper contributes four things:

  1. The control is run inside the official Graph-JEPA code at its published configurations, which reproduces its published accuracies to within 0.34 points. That part of the result therefore concerns the published model, not our re-implementation.

  2. The sign depends on the evaluation choice. With the read-out HP-JEPA publishes, the untrained encoder is significantly better on 3 of 5 datasets (ENZYMES −7.87, NCI1 −2.44, DD −3.48 points). That read-out is a fixed-strength probe on unscaled features, which Reviewer 5xmT rightly questions, so we read this as sensitivity to the probe, not as untrained features being better.

  3. Partition effects that a standard results table credits to pretraining are present at zero epochs. On NCI1 the SE-over-METIS gap is already +1.99 points with never-trained encoders (t_c 3.36), of +2.41 with trained ones. After submission we ran further controls under an exploratory protocol written before the runs; Section 3.3 (page 7) and Tables 16 and 17 report every contrast it registered, including those it did not support.

    • The SE fixed-K gap involves membership: SE fixed-K beats METIS with the same part sizes by +2.03 (t_c 3.18).
    • The SE-levels gap is consistent with its single-node rung, but that rung's own gain is not significant (+1.04, t_c 1.91).
    • The registered membership test for SE levels, its lead over the random partition, is not met either never-trained (+0.62, t_c 1.45) or trained (+0.68, t_c 0.87). Its lead over METIS with its part sizes is +0.69 (t_c 1.57) never-trained and +1.32 (t_c 2.011, at the threshold) trained.
    • Part sizes add nothing detectable beyond the node rung.

    Architectural claims of this kind can therefore be tested at zero epochs.

  4. Every null is reported as a bound, together with the smallest detectable effect.

Q3 / W2. A bound on a protocol known to be weak. We agree that the result is a bound, not parity (Section 3.4). It bounds what the published comparisons in this protocol can support: these methods report their accuracies and rank against each other here, by a median margin of 0.45 points, several times smaller than the smallest gain the protocol resolves (2.0 to 6.4). The bound is also not loose everywhere. On NCI1, the largest dataset (4,110 graphs), the upper end of the corrected 95% interval for the full-schedule gain is at most 1.7 points in every setting (Section 3.4, page 8).

Larger corpora, transfer and fine-tuning were not tested, and we do not claim the null extends to them; prior work finds that pretraining does help with large external corpora. The revision states this scope in the abstract and introduction (page 3), not only in Section 5.4. The abstract now says that pretraining adds no accuracy that these benchmarks can detect, and that gains of several points on a single small dataset are not excluded.

Q1. AI use and verification. We reviewed the AI-assisted code, records and text, and checked each finding of an adversarial review before accepting it, as the AI-use statement says. We do not claim to have re-derived every number by hand. The numbers can be checked in four ways:

  1. The trained official code reproduces the accuracies its own authors published, to within 0.34 points on all five datasets. This is an external anchor for the pipeline.
  2. Every number in the submitted paper is generated by released scripts from the released fold-level records. In the revision, Tables 14 and 15 are recomputed from those same records by new scripts. Tables 16 and 17 add runs made after submission, and Table 18 describes the partitions. These records and scripts are not yet in the supplementary material.
  3. A re-probed result was used only if the re-probe reproduced the stored accuracy exactly. It did on all 500 folds of the five datasets that could be reloaded; PROTEINS and IMDB-M failed that check and are excluded.
  4. The decision rules of the main tests were recorded before their runs, and each outcome is the one its rule gives (Appendix G).
Reviewer_5xmTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper asks a simple but important question: how much does pretraining actually contribute to the performance of graph JEPAs? The authors compare pretrained encoders with randomly initialized versions of the same architecture across seven TU datasets, using several partitioners, two read-outs, and two implementations. In most settings, they find no statistically significant improvement from pretraining. They also closely reproduce the official Graph-JEPA results, which makes the findings more convincing. The main recommendation is to routinely include untrained encoders as baselines and to be more careful when interpreting small benchmark differences.

The official-code reproduction is a strong point. The authors reproduce the published Graph-JEPA accuracies within 0.34 percentage points on all five evaluated datasets. Even with this close reproduction, pretraining significantly improves over random initialization only on MUTAG under the nominal test. This makes it harder to dismiss the findings as an implementation problem.

The novelty is somewhat limited. Earlier studies have already shown that untrained graph encoders can be competitive and recommended reporting them as baselines. The contribution here is mainly a detailed investigation of Graph-JEPA-style models. That is worthwhile, but the title and some statements suggest a broader conclusion than the experiments support. All datasets are small TU benchmarks, and the evaluation is limited to in-dataset pretraining with frozen linear probes.

Critical strengths

The official-code reproduction is a strong point. The authors reproduce the published Graph-JEPA accuracies within 0.34 percentage points on all five evaluated datasets. Even with this close reproduction, pretraining significantly improves over random initialization only on MUTAG under the nominal test. This makes it harder to dismiss the findings as an implementation problem.

Critical weaknesses
  1. The novelty is somewhat limited. Earlier studies have already shown that untrained graph encoders can be competitive and recommended reporting them as baselines. The contribution here is mainly a detailed investigation of Graph-JEPA-style models. That is worthwhile, but the title and some statements suggest a broader conclusion than the experiments support. All datasets are small TU benchmarks, and the evaluation is limited to in-dataset pretraining with frozen linear probes.

2.The probe configuration needs further examination. The experiments use fixed regularization settings, and the published read-out operates on unscaled features. If pretraining changes feature scale, the same regularization parameter may affect trained and untrained representations differently. This leaves some uncertainty about whether the results reflect representation quality or the probe settings. Feature standardization and nested tuning of regularization would help seperate these possibilities.

Questions for authors

The statistical conclusions could be more robust. Correcting for dependence between folds is reasonable, but the correction remains approximate. In the official implementation, the same folds are repeated across seeds. Averaging seeds within folds reduces the MUTAG statistic from 2.03 to 1.81, showing that this borderline finding is sensitive to the analysis. The pooled estimates also require care, a small common-effect estimate does not rule out meaningful gains on individual datasets.

2 comments
AuthorsOpenReview ↗

Every probe uses unscaled features at a fixed strength (now stated); stricter multiplicity checks; narrower claims

Thank you for the close reading. The revised PDF marks every change in blue.

W2. Probe settings. You are right, and the problem is wider than the submitted text said. Section 3.1 said that the published read-out fits a fixed-strength probe to unscaled features. So does our own read-out (strength 1), and so does the probe of the official Graph-JEPA code (its library default, also strength 1). No probe in the paper standardises its features or tunes its strength. If pretraining changes the feature scale, one fixed strength regularises the trained and untrained encoders differently.

The submitted record bears on this only indirectly. The two read-outs differ in strength by a factor of 100, and the trained encoder is significantly better under neither (0 of 7 and 0 of 5 datasets). They do disagree on the untrained encoder's lead, which is significant on 3 datasets only under the published read-out. They also read different rungs, so the comparison does not separate scale from content. A probe on standardised features, with a strength tuned inside each training fold, would separate them; it is not in this revision. The revision states the problem in Section 3.1 (page 4) and lists the missing probe as a limitation in Section 5.4 (page 10).

Q. Robustness of the statistics. We agree that the fold correction is approximate, and we read the MUTAG win as borderline.

  • Averaging the official code's seeds within each fixed fold lowers MUTAG from 2.03 to 1.81, as the paper reports. On the other four official datasets the same averaging leaves every statistic at most 0.93 in absolute value (Table 15).
  • Neither MUTAG win survives a Holm correction over the 29 one-sided comparisons; the smallest adjusted p is 0.53 (Section 3.1, page 5).
  • On pooling, we agree that a small common effect does not exclude larger gains on single datasets. Section 3.4 (page 8) and Table 14 now add a random-effects pooling, in which the gain may differ between datasets. Its 95% prediction interval for one further dataset reaches +3.27 points for the official code and +1.94 for SE levels. For the published read-out it reaches +3.99, around a pooled centre of −2.71. Per dataset, the upper ends of the corrected 95% intervals reach 7.9 points on MUTAG and 5.7 on ENZYMES, and are at most 3.2 on the other five; on NCI1, the largest benchmark, they are at most 1.7 in every setting. The abstract now says that gains of several points on a single small dataset are not excluded. Section 3.4 now gives the fixed-effect bound as 1.41 points, the official code's upper end, rather than a rounded 1.4.

W1. Novelty and scope. We agree that the untrained baseline is not new, and the paper says so (Sections 1 and 5.3). What it adds is narrower:

  1. The papers that propose patch-based graph JEPAs report no untrained comparison. We run it inside the official code at its published configurations, which reproduces its published accuracies to within 0.34 points.
  2. With the read-out that HP-JEPA publishes, a fixed-strength probe on unscaled features, the untrained encoder is significantly better on 3 of 5 datasets. Given W2, this shows how much the comparison depends on the evaluation choice, not that untrained features are better.
  3. Each null is reported with a bound and with the smallest effect the protocol can detect (2.0 to 6.4 points), set against the margins that rank published methods (median 0.45).
  4. A partition effect is split exactly into what the architecture gives before training and what pretraining adds.

We also agree that the title and some sentences read broader than the evidence. The abstract now names the setting: seven small TU benchmarks, in-dataset pretraining and a frozen linear probe. The introduction (page 3) now states that fine-tuning, transfer and larger pretraining corpora are not tested. The title is unchanged in this revision.

I appreciate your efforts on the revisions, but they are not substantial to me. So I will retain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.