- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The core idea of this paper is that existing Predictive Consistency Learning (PCL) for graph prediction may suffer from a shortcut solution. To address this issue, the paper proposes two main components: (1) a Contrastive Consistency Objective, which introduces cross-sample negative pairs to prevent trivial self-consistency collapse, and (2) Feature Perturbation, which breaks the identical conditioning across views and encourages the model to make better use of the noisy-label information. Extensive experiments are included to illustrate the effectiveness of the proposed method.
Critical strengths
- The paper identifies a meaningful shortcut issue in consistency-based graph prediction, and the motivation is clear.
- The experiments cover graph classification, node classification, and regression tasks, and also include multiple backbones.
Critical weaknesses
- The paper does not clearly establish why deterministic prediction should be considered a pathological shortcut. The main motivation assumes that ignoring the noisy label is harmful because it turns the generative predictor into a deterministic model, but this is not sufficiently justified for the prediction tasks studied in the paper.
- The proposed contrastive consistency objective is applied in the latent space, which does not necessarily guarantee consistency or generative behavior in the prediction space. The authors should further justify why this objective is a reasonable replacement for the original self-consistency objective.
- Most importantly, the empirical gains are generally quite small and do not convincingly support the claimed general effectiveness of the method. For example, in Table 1, GPS-GCCM improves over GPS-PCL by only 0.13 on MNIST and 0.077 on PATTERN, while the gains on PascalVOC-SP and COCO-SP are almost negligible. This makes the practical impact of the method less convincing.
- The ablation results do not convincingly show that the two proposed components are complementary. In Table 6, on CIFAR10, GPS+Consistency+Perturbation achieves 73.01, while GPS+GCCM achieves 72.50. This means the perturbation-only variant actually outperforms the full GCCM model by 0.51 points, suggesting that adding the contrastive objective does not consistently help on this benchmark.
Questions for authors
Please see the Weaknesses.
3 comments
W3. Performance improvements over PCL
We would like to clarify the following points:
-
The performance improvements appear small on some of these benchmarks because the compared methods themselves do not differ much on these benchmarks. For example, on MNIST the three strongest deterministic baselines in Table 1 (EGT, GRIT and GPS) differ from each other by less than 0.1 in accuracy, and our GCCM still achieves the best accuracy among all of them.
Model MNIST (Acc ↑) EGT 98.17 GRIT 98.11 GPS 98.08 GPS-PCL 98.11 GPS-GCCM (ours) 98.24 Similarly, our GCCM achieves the best F1 among all the compared methods on PascalVOC-SP and COCO-SP (Table 1).
-
Moreover, our GCCM aims to mitigate the shortcut solution that arises when applying consistency training to graph prediction (i.e., PCL), so the most direct comparison is with PCL. Below we provide the comparison between PCL and our GCCM under each of the three backbones (Table 8), where the better result in each pair is marked in bold.
Model MNIST (Acc ↑) CIFAR10 (Acc ↑) PATTERN (Acc ↑) CLUSTER (Acc ↑) ZINC (MAE ↓) SAN-PCL — — 86.768 77.461 0.140 SAN-GCCM (ours) — — 86.811 78.481 0.131 GPS-PCL 98.11 71.83 86.695 78.171 0.069 GPS-GCCM (ours) 98.24 72.50 86.772 78.820 0.069 GRIT-PCL 98.00 75.72 86.873 78.643 0.063 GRIT-GCCM (ours) 98.47 75.91 87.256 79.816 0.059 As shown in the table, by effectively mitigating the shortcut solution in consistency training, our GCCM outperforms PCL across different backbones and datasets, and leads to substantial performance improvements.
W4. Complementarity of the two components
We would like to clarify that the proposed contrastive consistency objective and feature perturbation do NOT mitigate the shortcut solution independently.
-
By introducing the additional separation requirement imposed by negative pairs, the contrastive consistency objective makes the shortcut solution generally no longer sufficient to minimize it (Lemma 2). However, its positive-pair agreement can still be trivially satisfied by the shortcut solution, since the conditioning input graph is identical at the two time steps and (Section 4.3).
-
Feature perturbation independently perturbs the input node features to break this identical conditioning input, so that the shortcut solution is in general no longer sufficient to guarantee the positive-pair agreement (Lemma 3), and thus further enhances the contrastive consistency objective.
-
Consistently, applying either component alone can NOT outperform GPS+Consistency on all the benchmarks (Table 6), and can NOT effectively mitigate the shortcut solution (Table 7). In contrast, combining both achieves better overall performance and effectively mitigates the shortcut solution (Fig. 3, Fig. 4 and Table 7).
Therefore, only applying BOTH of them can effectively mitigate the shortcut solution, which is consistent with the design of our GCCM.
Thank you for your response, which addressed some of my concerns. However, my primary concern remains unresolved: the improvement brought by GCCM is still quite limited. In particular, on MNIST, CIFAR-10, and PATTERN, the gains are marginal and do not convincingly demonstrate the effectiveness of the proposed method. I encourage the authors to evaluate their approach on additional datasets where GCCM may lead to more substantial improvements. Therefore, I will keep my current rating.
Thanks for providing constructive comments. Below we address each point in detail.
W1. Why the shortcut solution is harmful
We would like to clarify the following points:
The shortcut solution does NOT refer to the deterministic prediction at inference, but to the collapse of the denoising network in training. Stable sampling and one-step inference from a single noise sample are exactly what consistency training aims for (lines 50-54). Under the shortcut solution, however, the conditional generative model ignores (Yt,t) at every noise level during training, so the training objective reduces to a standard deterministic supervised loss (Eq. 22).
This collapse is harmful because the benefit of the consistency-based generative formulation is lost. With this formulation, the one-step prediction from pure noise, which is the hardest case, is learned together with easier cases: recovering the clean label is easy at low noise levels and becomes harder as the noise level increases, and the predictive consistency propagates the prediction accuracy from low-noise to high-noise conditions. Under this collapse, however, the model learns a direct mapping from the input graph to the target label at every noise level. Like a deterministic predictor, it then has to learn the hardest case directly, with no higher accuracy at low noise levels to propagate (please see our response to Reviewer N76b, W2 & Q2).
Moreover, consistent with this collapse, the experimental results show that vanilla consistency training in PCL collapses the model toward a deterministic predictor, while our GCCM, by mitigating the shortcut solution, outperforms PCL across different backbones and datasets (Table 8).
W2. Contrastive consistency objective in the latent space
We would like to clarify the following points:
Our contrastive consistency objective does NOT give up the consistency constraint, but only replaces the pairwise matching in the self-consistency objective, which admits the shortcut solution (Lemma 1). It retains the agreement between the two noise levels of the same instance through positive pairs, and additionally separates different instances through negative pairs, so that the shortcut solution is generally no longer sufficient to minimize it (Lemma 2). This is why it is a reasonable replacement for the self-consistency objective.
The latent representation and the prediction are obtained from the same encoder output through two projection heads (Eq. 4 and Section 4.2), so the agreement is enforced on the representation that the prediction relies on. In contrast, computing it directly in the label space may induce invariance and information compression on the predictions, which are optimized to satisfy the boundary condition objective, and thus interfere with its supervision (Section 4.2).
Moreover, the generative behavior of our GCCM in the prediction space is directly evaluated. Its prediction accuracy shows the clearest dependence on the informativeness of the injected label (Fig. 4), and it achieves the largest gains from (Yt,t) on all four benchmarks (Table 3).