acceptodds
Under review as a conference paper at ICLR 2027

Structural Neglect Matters: Overfitting and Its Mitigation in Graph Contrastive Learning

16 views

Abstract

Graph contrastive learning (GCL) has emerged as an effective framework for unsupervised graph representation learning, typically relying on GNN-based encoders to capture informative representations from both node attributes and graph structure. Despite the common expectation that optimizing contrastive objectives improves the discriminability of representations for downstream tasks, we reveal that excessive optimization of the graph contrastive objective can instead degrade the learned representations. Specifically, as optimization proceeds, the encoder progressively loses structural information, ultimately reducing the discriminability of the resulting representations for downstream tasks. We refer to this phenomenon as contrastive overfitting. Through theoretical analysis and extensive empirical investigation, we further identify the mechanism underlying contrastive overfitting and characterize how excessive contrastive optimization leads to the progressive degradation of structural information. Guided by these findings, we propose Progress–Adaptive REgulation (PARE), a simple yet effective adaptive training strategy that can be integrated into existing GCL methods to mitigate contrastive overfitting. Extensive experiments demonstrate the effectiveness of PARE across various GCL methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
unsupervised learning, graph representation learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qA1gOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing graph contrastive learning methods rely on optimizing contrastive objectives, but excessive optimization can degrade the learned representations. To investigate this problem, this paper theoretically analyzes the topology-independent optimal geometry of InfoNCE and the impact of structural degeneration on structure-dependent downstream tasks. Based on this, the authors propose Progress-Adaptive Regulation (PARE) to mitigate contrastive overfitting. Experiments across six datasets and nine GCL methods demonstrate the effectiveness of PARE in mitigating contrastive overfitting.

Critical strengths
  1. The identified problem is important and practically relevant. This paper investigates contrastive overfitting in graph contrastive learning and provides empirical evidence demonstrating that excessive contrastive optimization can degrade downstream performance.
  2. The experimental evaluation is extensive and demonstrates the effectiveness of PARE. The authors evaluate PARE across six datasets and nine InfoNCE-based GCL methods, including GRACE, GCA, PiGCL, ProGCL, and ReGCL. The experimental results show substantial improvements in downstream node classification performance, demonstrating the effectiveness of PARE in mitigating contrastive overfitting.
Critical weaknesses
  1. The theoretical connection between contrastive optimization and structural degeneration remains insufficiently established. The authors state that “The topology-independent optimum in Theorem 1 suggests that structural information may be lost during contrastive optimization.” However, Theorem 1 only characterizes the topology-independent optimal geometry of the InfoNCE objective, without establishing how approaching this optimum leads to the progressive loss of structural information during training. Therefore, the connection between the topology-independent optimum and structural degeneration remains unclear, and the theoretical analysis does not sufficiently support the claimed mechanism of contrastive overfitting.

  2. Insufficient alignment between the identified problem and the proposed method. The paper attributes contrastive overfitting to the lack of explicit structural preservation in the InfoNCE objective. However, the proposed PARE method does not directly incorporate graph structural information or impose structural preservation constraints. Instead, it regulates parameter updates based solely on contrastive loss reduction and representation drift. Therefore, it remains unclear whether PARE genuinely addresses structural neglect or merely alleviates performance degradation by suppressing excessive optimization.

  3. The evaluation of structural dependence is limited and does not fully support the generality of Theorem 2. The authors define structural dependence in Equation (3) as the label information provided by graph structure beyond node attributes. However, in the experiments on real-world datasets, they use the supervised performance gap between GCN and MLP as a proxy for structural dependence. These two quantities are not equivalent. Furthermore, although the paper discusses structure-dependent downstream tasks, the experimental evaluation is limited to node classification. The authors should consider additional downstream tasks that strongly depend on graph structure to further validate the generality of their conclusions.

  4. The evaluation protocol may overstate the practical benefits of PARE. The authors train all baseline methods for 10,000 epochs. However, some methods, such as GRACE, may already achieve favorable node classification performance after 500 or 1,000 epochs. Continuing training until 10,000 epochs may amplify the observed performance degradation. The authors should provide additional comparisons using the original training configurations of the baseline methods to better assess the practical effectiveness of PARE.

Questions for authors
  1. Could the authors further clarify how the topology-independent optimal geometry characterized in Theorem 1 leads to the progressive loss of structural information during contrastive optimization? What theoretical evidence supports this connection?
  2. PARE regulates parameter without explicitly incorporating graph structural information, could the authors clarify how PARE mitigates structural degeneration rather than merely suppressing excessive optimization?
  3. Why are all baseline methods trained for 10,000 epochs rather than following their original training configurations? Could the authors provide comparisons using the recommended training schedules or appropriate early-stopping strategies to better demonstrate the practical effectiveness of PARE?
Reviewer_7oZ1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies contrastive overfitting in graph contrastive learning, where continued optimization of the contrastive objective reduces training loss but degrades downstream performance. The authors argue that instance-level InfoNCE favors a topology-independent representation geometry that separates individual nodes without explicitly preserving their structural relationships. They further characterize the impact of this behavior through the structural dependence of a downstream task, defined as Δstr=I(Y;S∣U)\Delta_{\mathrm{str}}=I(Y;S\mid U). Both synthetic and real-data experiments show that tasks relying more strongly on graph structure experience larger performance degradation. Based on the observed transition from high-return to low-return optimization, the paper proposes Progress-Adaptive Regulation (PARE). PARE dynamically scales the update strength according to the contrastive improvement and representation drift. Experiments across several GCL methods and six datasets show that PARE generally improves final-checkpoint performance and reduces sensitivity to training duration and learning rate.

Critical strengths

The paper identifies an important mismatch between contrastive optimization and downstream representation quality. The combination of the regular-simplex analysis, controlled SBM experiments, and structural decodability measurements provides a coherent explanation of why continued InfoNCE optimization can harm graph-dependent tasks.

PARE is simple and broadly applicable. It does not require downstream labels or changes to the encoder, augmentation strategy, or contrastive objective.

The empirical analysis goes beyond final accuracy. The paper tracks loss, downstream accuracy, representation drift, and structural decodability throughout training. This provides stronger evidence for the proposed mechanism than a comparison based only on final checkpoints.

Critical weaknesses

It is unclear whether PARE specifically mitigates structural degeneration or mainly acts as an adaptive learning-rate schedule. PARE only uses contrastive improvement and representation drift, without directly measuring graph structure. Stronger comparisons with carefully tuned learning-rate decay and unsupervised early-stopping strategies are needed to isolate its contribution.

The scalability of the representation-drift calculation is not sufficiently evaluated. Computing Ct=∣Zt−Zt−1∣F/NC_t=|Z_t-Z_{t-1}|_F/\sqrt{N} requires storing and comparing node representations across iterations. This is manageable for the evaluated full-batch graphs, but its cost and reliability are unclear for million-node graphs or mini-batch training.

The real-data measure of structural dependence is only a proxy. Using the performance gap between a GCN and an MLP may also reflect differences in model capacity, optimization, or regularization, rather than only I(Y;S∣U)I(Y;S\mid U). This makes the empirical relationship between structural dependence and overfitting less conclusive on real datasets.

Questions for authors

How does PARE compare with carefully tuned cosine decay, ReduceLROnPlateau, warmup with decay, unsupervised early stopping, and parameter or representation EMA? Can the authors clarify whether its gains arise specifically from mitigating structural degeneration?

How can the representation-drift term be computed efficiently in million-node graphs or mini-batch training, where consecutive batches may contain different nodes? Please also report its memory and runtime overhead.

Can the authors further validate the GCN–MLP performance gap as a proxy for I(Y;S∣U)I(Y;S\mid U)? How sensitive are the conclusions to differences in model capacity, optimization, and regularization between the two models?

Reviewer_HbU7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the overfitting issue in graph contrastive learning. The authors observe that the over-optimization of InfoNCE-based contrastive losses can impair performance on downstream tasks. To address this, they propose PARE, a simple and effective plug-in for GCL. The authors conducted extensive experiments.

Critical strengths

S1. The motivation is clear and interesting. Through experimental observation, the authors point out that excessive optimization of the graph contrastive objective can, in fact, degrade the learned representations.

S2. The proposed method is simple and effective.

S3. I appreciate the writing in this paper.

Critical weaknesses

I have the following concerns and would like to discuss them with the authors.

W1. The method proposed by the authors indeed yields a significant improvement in accuracy compared to the "Final" baseline. My main concern, however, is that "+PARE" does not offer a substantial gain over the "Oracle" baseline. For instance, if a model trained for 10,000 iterations achieves lower accuracy than one trained for only 200 iterations, then why bother with 10,000 iterations at all? This is my main concern. I believe that if this point can be well explained, it will enhance the contribution of this paper.

W2. The compared baselines are primarily from works published before 2024. It would be beneficial to include some more recent contrastive algorithms, such as [1], [2], [3].

W3. The methodology section is rather brief. It would be better to include a simple model diagram or an algorithm flowchart.

[1] Does GCL need a large number of negative samples? Enhancing graph contrastive learning with effective and efficient negative sampling. AAAI, 2025.

[2] CaliGCL: Calibrated Graph Contrastive Learning via Partitioned Similarity and Consistency Discrimination. NeurIPS, 2025.

[3] Revisiting Positive Samples in Graph Contrastive Learning: From the Perspective of Message Passing. ICML, 2026.

Questions for authors

Please refer to the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.