- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper compares pretrained and randomly initialized graph JEPA encoders on small TU datasets to evaluate pretraining gains in linear-probe classification accuracy. Pretraining changes the representations, but significant accuracy gains are detected only in a few settings, and some partitioning advantages exist before training. Through official code reproduction, controlled experiments, and corrected statistical tests, the paper separates pretraining gains from architectural contributions and quantifies the benchmarks’ ability to detect small differences in accuracy.
Critical strengths
- The paper is clearly structured and easy to follow.
- Comparing with the untrained encoder helps to distinguish architectural contributions from pretraining gains. The research question is relevant to the evaluation of graph self-supervised learning.
- The experimental design is well-structured, covering multiple datasets, partitions, and readout configurations, providing different support for the problem. It also accounts for dependencies between cross-validation folds, reporting uncertainty and detection capability.
Critical weaknesses
- The paper reports dataset-level averages without analyzing which types of samples benefit from or are adversely affected by pretraining. It is unclear whether small overall differences mask opposing effects across samples.
- The paper does not analyze the relationship between graph size and partition granularity. It remains untested whether partitioning small graphs produces overly small or highly overlapping subgraphs that provide insufficient useful context for pretraining.
- The paper does not examine how the pretraining prediction objective relates to the downstream probe objective or whether the learned representations suit classification. It can also compare the joint fine-tuning of the encoder and the classifier, and compare the gradient directions and magnitudes of the two objectives at the same encoder state.
- Despite the partition controls and decomposition of performance differences, the paper does not sufficiently explain why random encoders perform strongly or why pretraining yields limited gains.
Questions for authors
See the weakness.
2 comments
I thank the reviewer for their responses. I maintain my original (generally positive) assessment of the paper.
Patch size against graph size, never-trained partition controls, and what the records cannot show
Thank you for the constructive review. The revised PDF marks every change in blue.
W4. Why untrained encoders are strong. Section 3.3 gives our reading. Random-walk features pooled within a partition, passed through randomly weighted GINE layers, already give a linear probe strong structural statistics. If that is right, partition effects should be visible before any training. After submission we tested this under an exploratory protocol written before the runs. Section 3.3 (page 7) and Tables 16 and 17 report every contrast it registered, including those it did not support.
So the partition effects on NCI1 are present before training, which supports the architectural reading, though not every step of the attribution is significant. Neither the uniformity of part sizes nor the share of ring bonds cut predicted the gap across eight datasets; all four registered rank correlations were negative. What in the partition helps remains open.
Why pretraining adds so little, we cannot fully explain. One candidate is that the prediction task of our backbone is easy, because each normalised target scalar is confined to a small range (Section 2). But the official code, which has no such bound, gives the same verdict.
W2. Graph size and partition granularity. We measured patch sizes directly from the partitions, with no training (Table 18). The METIS ladder uses cluster counts up to 12 on every dataset except IMDB-M, where they run from 2 to 6. The finest rung averages 1.5 nodes per part on MUTAG, 1.6 on IMDB-B, 2.2 on IMDB-M, 2.5 on NCI1, 2.8 on ENZYMES, 3.4 on PROTEINS and 24.7 on DD (DD measured on every tenth graph). Each part is then expanded by one hop.
So you are right that the finest patches on small graphs are nearly single nodes. Larger parts do not go with larger pretraining gains, though. DD has by far the largest parts, and with METIS partitions under our read-out it has the most negative difference (−1.41). Over the seven datasets, the rank correlation between nodes per part and the METIS gain is −0.54. That is seven points and exploratory, so it argues against small patches as the main cause without ruling them out. We did not measure patch overlap. Section 5.4 (page 10) now says both.
W1. Per-sample effects. We agree this would be informative. Per-graph predictions were not stored. They could be regenerated from the saved fold models on the five datasets whose models reload exactly (not PROTEINS or IMDB-M), but we could not complete this within the discussion period. The fold records show only that the sign varies across seeds: with METIS partitions the trained encoder is ahead in 4 of 5 seeds on NCI1 and in 0 of 5 on DD.
W3. The pretraining objective and the probe. We did not examine this; joint fine-tuning and a comparison of the two objectives' gradients were not run. Section 3.2 shows only that pretraining moves the representation further from its initialisation than two initialisations are from each other, without improving what the probe reads. That exploratory result is consistent with the objective changing the features in directions a linear probe does not use, but it does not show it. Section 5.4 now names joint fine-tuning and the gradient comparison as the next tests.
Prompted by the other reviews, the revision also adds: