acceptodds
Under review as a conference paper at ICLR 2027

GCCM: Enhancing Generative Graph Prediction via Contrastive Consistency Model

Abstract

Conditional generative models, particularly diffusion-based methods, have recently been applied to graph prediction by modeling the target as a conditional distribution given the input graph, yielding competitive results compared to deterministic predictor. However, existing diffusion-based prediction methods typically require expensive iterative denoising at inference and often suffer from unstable sampling, which motivates recent efforts to reduce inference denoising steps and enable stable sampling via techniques such as consistency training. Despite this progress, we find that existing consistency training methods for graph prediction could potentially fall into a shortcut solution: the model may attempt to satisfy the self-consistency constraint by ignoring the noisy target (i.e., assigning it negligible weight), ultimately collapsing into a purely deterministic predictor. To mitigate such shortcut solution, we propose GCCM, a graph contrastive consistency model that goes beyond isolated pairwise matching between the same target at different noise levels by introducing negative pairs into a contrastive consistency objective. This adds an additional separation requirement, making the shortcut solution no longer trivially sufficient to satisfy the proposed objective. Moreover, we apply feature perturbation to the input node/edge features to break identical conditioning on the input graph, so that the shortcut no longer yields the same predictions across noise levels and becomes less attractive. Extensive experiments on benchmark datasets demonstrate that GCCM mitigates the shortcut solution and yields consistent performance improvements in graph prediction compared to deterministic predictors.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph representation learning, Generative diffusion models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pdxAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The core idea of this paper is that existing Predictive Consistency Learning (PCL) for graph prediction may suffer from a shortcut solution. To address this issue, the paper proposes two main components: (1) a Contrastive Consistency Objective, which introduces cross-sample negative pairs to prevent trivial self-consistency collapse, and (2) Feature Perturbation, which breaks the identical conditioning across views and encourages the model to make better use of the noisy-label information. Extensive experiments are included to illustrate the effectiveness of the proposed method.

Critical strengths
  • The paper identifies a meaningful shortcut issue in consistency-based graph prediction, and the motivation is clear.
  • The experiments cover graph classification, node classification, and regression tasks, and also include multiple backbones.
Critical weaknesses

  • The paper does not clearly establish why deterministic prediction should be considered a pathological shortcut. The main motivation assumes that ignoring the noisy label YtY_t is harmful because it turns the generative predictor into a deterministic model, but this is not sufficiently justified for the prediction tasks studied in the paper.
  • The proposed contrastive consistency objective is applied in the latent space, which does not necessarily guarantee consistency or generative behavior in the prediction space. The authors should further justify why this objective is a reasonable replacement for the original self-consistency objective.
  • Most importantly, the empirical gains are generally quite small and do not convincingly support the claimed general effectiveness of the method. For example, in Table 1, GPS-GCCM improves over GPS-PCL by only 0.13 on MNIST and 0.077 on PATTERN, while the gains on PascalVOC-SP and COCO-SP are almost negligible. This makes the practical impact of the method less convincing.
  • The ablation results do not convincingly show that the two proposed components are complementary. In Table 6, on CIFAR10, GPS+Consistency+Perturbation achieves 73.01, while GPS+GCCM achieves 72.50. This means the perturbation-only variant actually outperforms the full GCCM model by 0.51 points, suggesting that adding the contrastive objective does not consistently help on this benchmark.
Questions for authors

Please see the Weaknesses.

3 comments

Thanks for providing constructive comments. Below we address each point in detail.


W1. Why the shortcut solution is harmful

We would like to clarify the following points:

  • The shortcut solution does NOT refer to the deterministic prediction at inference, but to the collapse of the denoising network in training. Stable sampling and one-step inference from a single noise sample are exactly what consistency training aims for (lines 50-54). Under the shortcut solution, however, the conditional generative model ignores (Yt,t)(\mathbf{Y}_t, t) at every noise level during training, so the training objective reduces to a standard deterministic supervised loss (Eq. 22).

  • This collapse is harmful because the benefit of the consistency-based generative formulation is lost. With this formulation, the one-step prediction from pure noise, which is the hardest case, is learned together with easier cases: recovering the clean label is easy at low noise levels and becomes harder as the noise level increases, and the predictive consistency propagates the prediction accuracy from low-noise to high-noise conditions. Under this collapse, however, the model learns a direct mapping from the input graph to the target label at every noise level. Like a deterministic predictor, it then has to learn the hardest case directly, with no higher accuracy at low noise levels to propagate (please see our response to Reviewer N76b, W2 & Q2).

  • Moreover, consistent with this collapse, the experimental results show that vanilla consistency training in PCL collapses the model toward a deterministic predictor, while our GCCM, by mitigating the shortcut solution, outperforms PCL across different backbones and datasets (Table 8).


W2. Contrastive consistency objective in the latent space

We would like to clarify the following points:

  • Our contrastive consistency objective does NOT give up the consistency constraint, but only replaces the pairwise matching in the self-consistency objective, which admits the shortcut solution (Lemma 1). It retains the agreement between the two noise levels of the same instance through positive pairs, and additionally separates different instances through negative pairs, so that the shortcut solution is generally no longer sufficient to minimize it (Lemma 2). This is why it is a reasonable replacement for the self-consistency objective.

  • The latent representation and the prediction are obtained from the same encoder output through two projection heads (Eq. 4 and Section 4.2), so the agreement is enforced on the representation that the prediction relies on. In contrast, computing it directly in the label space may induce invariance and information compression on the predictions, which are optimized to satisfy the boundary condition objective, and thus interfere with its supervision (Section 4.2).

  • Moreover, the generative behavior of our GCCM in the prediction space is directly evaluated. Its prediction accuracy shows the clearest dependence on the informativeness of the injected label (Fig. 4), and it achieves the largest gains from (Yt,t)(\mathbf{Y}_t, t) on all four benchmarks (Table 3).


W3. Performance improvements over PCL

We would like to clarify the following points:

  • The performance improvements appear small on some of these benchmarks because the compared methods themselves do not differ much on these benchmarks. For example, on MNIST the three strongest deterministic baselines in Table 1 (EGT, GRIT and GPS) differ from each other by less than 0.1 in accuracy, and our GCCM still achieves the best accuracy among all of them.

    ModelMNIST (Acc ↑)
    EGT98.17
    GRIT98.11
    GPS98.08
    GPS-PCL98.11
    GPS-GCCM (ours)98.24

    Similarly, our GCCM achieves the best F1 among all the compared methods on PascalVOC-SP and COCO-SP (Table 1).

  • Moreover, our GCCM aims to mitigate the shortcut solution that arises when applying consistency training to graph prediction (i.e., PCL), so the most direct comparison is with PCL. Below we provide the comparison between PCL and our GCCM under each of the three backbones (Table 8), where the better result in each pair is marked in bold.

    ModelMNIST (Acc ↑)CIFAR10 (Acc ↑)PATTERN (Acc ↑)CLUSTER (Acc ↑)ZINC (MAE ↓)
    SAN-PCL——86.76877.4610.140
    SAN-GCCM (ours)——86.81178.4810.131
    GPS-PCL98.1171.8386.69578.1710.069
    GPS-GCCM (ours)98.2472.5086.77278.8200.069
    GRIT-PCL98.0075.7286.87378.6430.063
    GRIT-GCCM (ours)98.4775.9187.25679.8160.059

    As shown in the table, by effectively mitigating the shortcut solution in consistency training, our GCCM outperforms PCL across different backbones and datasets, and leads to substantial performance improvements.


W4. Complementarity of the two components

We would like to clarify that the proposed contrastive consistency objective and feature perturbation do NOT mitigate the shortcut solution independently.

  • By introducing the additional separation requirement imposed by negative pairs, the contrastive consistency objective makes the shortcut solution generally no longer sufficient to minimize it (Lemma 2). However, its positive-pair agreement can still be trivially satisfied by the shortcut solution, since the conditioning input graph is identical at the two time steps t1t_1 and t2t_2 (Section 4.3).

  • Feature perturbation independently perturbs the input node features to break this identical conditioning input, so that the shortcut solution is in general no longer sufficient to guarantee the positive-pair agreement (Lemma 3), and thus further enhances the contrastive consistency objective.

  • Consistently, applying either component alone can NOT outperform GPS+Consistency on all the benchmarks (Table 6), and can NOT effectively mitigate the shortcut solution (Table 7). In contrast, combining both achieves better overall performance and effectively mitigates the shortcut solution (Fig. 3, Fig. 4 and Table 7).

Therefore, only applying BOTH of them can effectively mitigate the shortcut solution, which is consistent with the design of our GCCM.

Thank you for your response, which addressed some of my concerns. However, my primary concern remains unresolved: the improvement brought by GCCM is still quite limited. In particular, on MNIST, CIFAR-10, and PATTERN, the gains are marginal and do not convincingly demonstrate the effectiveness of the proposed method. I encourage the authors to evaluate their approach on additional datasets where GCCM may lead to more substantial improvements. Therefore, I will keep my current rating.

Reviewer_N76bOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors use a consistency model for graph prediction tasks. They find that the model learns to ignore the noisy input and becomes a deterministic predictor. To fix this, they replace the consistency loss with a contrastive one and perturb the input features.

Critical strengths

None

Critical weaknesses

I have two major concerns for which I recommend clear rejection of the paper.

  1. The central method is flawed. The authors train a consistency model, but the loss they optimize is not a consistency loss. A consistency model (Song et al., 2023) matches the outputs of the consistency function at two points on the same probability-flow ODE trajectory. These points come either from a pretrained diffusion model (consistency distillation) or from the same noise sample at two adjacent noise levels (consistency training). This paper instead noises the label twice, independently, at noise levels tt and 0.2t0.2t (Algorithm 1). Because the two noise samples are independent, the two outputs can only match if neither depends on its input. The paper also drops the skip connection that is essential to consistency models to enforce the boundary condition at t=0t = 0. So the "shortcut" that the paper identifies and aims to solve as its core contribution is just a consequence of an incorrect implementation of consistency models.

  2. The generative formulation is not motivated. For the reported metrics (accuracy, MAE, F1), the optimal predictor is a deterministic function of the input graph. So it is not clear why modeling a distribution, with a consistency model or any other generative method, should help. The authors neither provide this motivation nor evaluate any distributional property. Prior work does not supply this motivation either. Most of the cited diffusion work on graphs concerns different tasks, such as graph generation (e.g., DiGress). The only cited graph prediction works that use diffusion are LGD and PCL. LGD motivates generative prediction with unifying generation and prediction and with uncertanty estimation; but this paper uses neither. PCL uses noisy labels to improve optimization and to refine predictions, but it does not model a distribution over labels.

There are further issues. The loss is taken from PCL, which deliberately aims for predictions that do not depend on the noisy label, so the paper treats PCL's intended behavior as a failure. The proposed contrastive loss does not penalize the shortcut either: Lemma 2 only covers a degenerate case, and Sec. 4.3 itself admits that the shortcut still aligns positive pairs perfectly. Finally, the empirical gains are small and often within one standard deviation.

Questions for authors
  1. Why do the authors sample the noise independently instead of using the proper consistency loss? Does the "shortcut solution" still occur with the proper consistency loss?
  2. What is the benefit of modeling a label distribution for these tasks?
4 comments

Thanks for providing constructive comments. Below we address each point in detail.


W1 & Q1. Consistency training for graph prediction and the shortcut solution

W1.1 & Q1.1. Independent sampling of the two noisy labels

We would like to clarify why we sample the two noisy labels independently instead of using the consistency loss of consistency models [1].

  • The consistency loss in [1] is designed for generation, where different trajectories start from different data points, so the self-consistency is only enforced within each trajectory.

  • In prediction settings, each input graph is paired with a single reference target label, so the model trades output diversity for capturing this target label, and the predictive consistency has to be enforced across all label noising trajectories. This is exactly the predictive consistency framework of PCL (Sec. 4.3), which samples the two noisy labels independently in both time steps and trajectories, and the same definition is adopted by other methods, such as Fast T2T [2] (Def. 4.1).

Since our GCCM aims to improve PCL, our Eq. 5 and Algorithm 1 follow its formulation, which is NOT an incorrect implementation, but the consistency training for graph prediction in PCL.

W1.2. Skip connection and boundary condition

We would like to clarify that the boundary condition is NOT dropped without the skip connection, but enforced by the boundary condition objective, which supervises the predictions at both noise levels toward the target label (Eq. 5). As explained in Section 4.1, the additive skip-connection parameterization assumes that the clean labels and the model output lie in the same continuous space, which is less natural for discrete class labels, so PCL enforces the boundary condition through an explicit supervised objective instead.

W1.3 & Q1.2. Shortcut solution with a shared noise sample

We would like to clarify that the shortcut solution still exists when the two noisy labels share the same noise sample as in [1], so it is NOT a consequence of independent sampling. Lemma 1 does not depend on how the two noisy labels are sampled: under Fsc\mathcal{F}_{\mathrm{sc}}, the two predictions are identical regardless of the noisy labels, so they also match when the two noisy labels share the same noise sample. The reviewer's observation that "the two outputs can only match if neither depends on its input" is exactly this shortcut solution, which arises in the predictive consistency framework of PCL and motivates our GCCM to modify the consistency objective itself (Section 4.2).

[1] Consistency Models.
[2] Fast T2T: Optimization Consistency Speeds Up Diffusion-Based Training-to-Testing Solving for Combinatorial Optimization.

W2 & Q2. Motivation of the generative formulation

We would like to clarify that the key difference between a deterministic predictor and a consistency-based generative predictor is whether the input contains noise.

  • Without noise, a deterministic predictor has to learn to directly map the input graph to the label in one step, which can be difficult to learn.

  • With noise, a generative predictor takes an additional noisy input, which is pure noise at inference. During training, this noisy input is a noisy label obtained by corrupting the ground-truth label at different noise levels, and the predictor learns to map it back to the ground-truth label conditioned on the input graph.

  • The benefit of this learning paradigm is that the prediction can be learned from easy to hard across noise levels. At low noise levels, the noisy label is close to the ground-truth label, so the mapping is easy to learn and its prediction is accurate; as the noise level increases, the noisy label becomes less informative and the mapping becomes harder, up to pure noise, which is exactly the input at inference. In this way, the easier cases at low noise levels help the generative predictor learn the harder ones, up to the prediction from pure noise. Since pure noise carries no information about the label, this hardest case is the same as the prediction of a deterministic predictor, but it is now learned together with the easier cases.

  • Consistently, when the noisy label is injected at different noise levels, our GCCM generally achieves the highest prediction accuracy among the model variants at low and intermediate noise levels (Fig. 4), showing that it indeed utilizes the noisy label in the easier cases. Moreover, with the same backbone, our GCCM outperforms the deterministic predictor GPS on all seven benchmarks (Table 1).


W3. PCL's intended behavior and the shortcut solution

We would like to clarify the following points:

  • PCL's goal of making predictions rely on Yt\mathbf{Y}_t as little as possible concerns the inference, which enables stable sampling and one-step inference from a single noise sample (lines 50-54). In training, PCL learns this one-step prediction together with easier low-noise cases, and its predictive consistency propagates the prediction accuracy from low-noise to high-noise conditions, as clarified in our response to W2 & Q2.

  • The shortcut solution is the collapse of the denoising network in training, rather than the stable sampling and one-step inference intended by PCL. Under the shortcut solution, (Yt,t)(\mathbf{Y}_t, t) is ignored at every noise level during training, so the prediction at low noise levels is no longer easier, there is no higher accuracy to propagate, and this benefit of PCL is lost. Consistently, on MNIST, even a highly informative Yt\mathbf{Y}_t brings only limited improvement to GPS+Consistency (Fig. 4). Our GCCM mitigates the shortcut solution (Table 3 and Fig. 4), while keeping the stable sampling and one-step inference from a single noise sample.


W4. Role of the lemmas in our argument

Please see our response to Reviewer AUoq (W2.2 and W2.3).


W5. Performance improvements

Please see our response to Reviewer AUoq (W1.1).

I thank the authors for the response.

I think the main issue here (as I said in my original review) is that the paper applies a generative model to a task that is inherently not generative. The response confirms this. It says that each graph has a single target label and that predicting from pure noise is the same as what a deterministic predictor does. This contradicts the paper’s motivation of modeling a distribution over labels.

Every problem the paper tries to fix comes from using a generative model for this task. With a single correct label per input, there is nothing to sample, and a model that ignores the noise is doing the right thing.

The response now argues that noisy labels let the model learn from easy cases to hard ones (which, by the way, is a completely different motivation from the one in the paper, so if this were the case, the paper would need to be revised). I don’t find that argument convincing. If it held, it would apply to any machine learning task, as you can always give the model a partially corrupted answer during training. And even here, the model still needs to predict the answer from pure noise at inference, and must also learn consistency across noise levels.

I can imagine that with enough tuning the model beats some baselines, and that is what the results show. But these benchmarks are saturated, and the gains are small and often within one standard deviation. This does not show that the generative formulation helps.

For these reasons, I keep my original rating.

Thanks for your further comments.


W1. Motivation and benefit of the generative formulation

We would like to clarify the following points:

  • Learning the prediction from easy to hard does NOT contradict the motivation of modeling a distribution over labels, but is a benefit of the way this distribution is modeled by diffusion-based generative models. This benefit was not stated clearly in the original submission, and we have revised the Abstract and the Introduction to state it explicitly (highlighted in blue).

  • Our previous statement that predicting from pure noise is the same as deterministic prediction refers to its difficulty, not to how it is learned. At inference, diffusion-based generative models predict the label by denoising from pure noise. Since pure noise retains no label information, this prediction is as difficult as deterministic prediction. However, during training, these models corrupt the ground-truth label at different noise levels up to pure noise and learn to recover the ground-truth label from the noisy label, conditioned on the input graph. This prediction is thus learned jointly with the easier cases at lower noise levels, where the noisy label retains more label information. Instead of being learned only at its hardest, which can be difficult to learn, the prediction is learned from easy to hard.

Therefore, this benefit can be obtained in a prediction task by formulating the task as conditional generation, as we do for graph prediction.


W2. Shortcut solution in consistency training

We would like to clarify that the shortcut solution ignores the noisy label during training, not only at inference. Applying consistency training to graph prediction admits a shortcut solution, under which Wy=0W_y = 0 and Wt=0W_t = 0 (Lemma 1). These weights are shared across all noise levels (Eq. 3), so the noisy label is ignored at every noise level during training, including the low noise levels where it is close to the ground-truth label. The training objective then reduces to a standard deterministic supervised loss (Eq. 22). The denoising network has to learn only the hardest case, losing the benefit of learning graph prediction from easy to hard across noise levels.

Therefore, our GCCM aims to mitigate the shortcut solution and retain this benefit.


W3. Performance improvements

We would like to clarify that we follow LGD in using GPS as the backbone, so that LGD, PCL and our GCCM are compared under the same backbone for a fair comparison (Section 5.1). With the same backbone, our GCCM, which takes the noisy label as an additional input, outperforms the deterministic predictor GPS on all seven benchmarks (Table 1). Moreover, with SAN and GRIT as two additional backbones, our GCCM consistently outperforms LGD and PCL under the same backbone (Table 8).

Reviewer_AUoqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates consistent training for graph prediction (based on diffusion). The researchers found that existing methods suffer from “step-level inefficiency” and “sample-level inefficiency.” To address these issues, they propose GCCM, which has the following features: (1) It employs a contrastive consistency objective function in the latent space, treating mini-batches of instances as negative samples; (2) It applies independent feature perturbations to the input graph across two iterative processes. The study compared GCCM with LGD and PCL on seven graph benchmark datasets using GPS, SAN, and GRIT backbone networks. Diagnostic ablation experiments demonstrated that GCCM enhances its reliance on noisy labels while maintaining single-step inference.

Critical strengths
  • This paper identifies a specific and plausible failure mode in previous research and provides direct evidence to support it. The authors found that by setting the noisy labels and time-step projections to zero, the model can be simplified to a deterministic predictor that satisfies PCL-style self-consistency. This approach appears simple yet compelling. Importantly, this conclusion is supported by direct diagnostic results, which indicate that standard consistency training exhibits the lowest dependence on (Y_t, t). This point is crucial to my evaluation, as this diagnosis constitutes the paper’s primary academic contribution; even if the proposed remedy is only partially effective, this diagnosis remains of significant value to the academic community.
  • The ablation experiments are well-documented and reliable, including “consistency + perturbation” variants that isolate each component. This is important because it lends credibility to the reported data and makes the method easy to adopt, which to some extent compensates for the relatively modest improvement.
Critical weaknesses
  • The performance improvement is modest, and ablation experiments indicate that the issue lies in the feature perturbations rather than the comparison objective function. CLUSTER is the only method that shows a significant improvement. Table 6 shows that the comparison term alone has almost no effect, whereas using perturbations alone performs comparably to or better than the full GCCM on multiple benchmarks. Since perturbations are essentially standard input noise enhancement, a deterministic baseline of GPS plus the same enhancement is required. However, this baseline is missing from the paper.

  • These lemmas do not prove what is required for the argument. Lemma 1 merely shows that the “shortcut” is a sufficient solution. Lemma 2 proves its insufficiency by additionally setting Wx=0W_x = 0, but this merely demonstrates a suboptimal solution rather than proving that the set is suboptimal; Section 4.3 subsequently acknowledges that negative samples can be separated using XX alone. The theoretical section provides only a framework, not evidence.

Questions for authors
  • Is it possible to add a deterministic GPS that is trained using the same feature perturbations as data augmentation, without diffusion or consistency training?
  • When λ2\lambda_2 is reduced, does the shortcut in PCL still exist? If a smaller consistency weight has already restored the dependence on (Yt,t)(Y_t, t), then the proposed objective function may no longer be necessary.
  • Since Y_T is purely noise during the inference stage, through what mechanism does greater reliance on (YtY_t, t) improve test accuracy? Is averaging multiple Y_T samples effective?
4 comments

Thanks for providing constructive comments. Below we address each point in detail.


W1 & Q1. Performance improvements and the roles of the two components

W1.1. Performance improvements

We would like to clarify the following points:

  • The performance improvements appear modest on some of these benchmarks because the compared methods themselves do not differ much on these benchmarks. For example, on MNIST the three strongest deterministic baselines in Table 1 (EGT, GRIT and GPS) differ from each other by less than 0.1 in accuracy, and our GCCM still achieves the best accuracy among all of them.

    ModelMNIST (Acc ↑)
    EGT98.17
    GRIT98.11
    GPS98.08
    GPS-PCL98.11
    GPS-GCCM (ours)98.24
  • Moreover, our GCCM aims to mitigate the shortcut solution that arises when applying consistency training to graph prediction (i.e., PCL), so the most direct comparison is with PCL. Below we provide the comparison between PCL and our GCCM under each of the three backbones (Table 8), where the better result in each pair is marked in bold.

    ModelMNIST (Acc ↑)CIFAR10 (Acc ↑)PATTERN (Acc ↑)CLUSTER (Acc ↑)ZINC (MAE ↓)
    SAN-PCL——86.76877.4610.140
    SAN-GCCM (ours)——86.81178.4810.131
    GPS-PCL98.1171.8386.69578.1710.069
    GPS-GCCM (ours)98.2472.5086.77278.8200.069
    GRIT-PCL98.0075.7286.87378.6430.063
    GRIT-GCCM (ours)98.4775.9187.25679.8160.059

    As shown in the table, by effectively mitigating the shortcut solution in consistency training, our GCCM outperforms PCL across different backbones and datasets, and leads to substantial performance improvements.

W1.2. Roles of the two components

We would like to clarify that the proposed contrastive consistency objective and feature perturbation do NOT serve as independent remedies.

  • By introducing the additional separation requirement imposed by negative pairs, the contrastive consistency objective makes the shortcut solution generally no longer sufficient to minimize it (Lemma 2). However, its positive-pair agreement can still be trivially satisfied by the shortcut solution, since the conditioning input graph is identical at the two time steps t1t_1 and t2t_2 (Section 4.3).

  • Feature perturbation independently perturbs the input node features to break this identical conditioning input, so that the shortcut solution is in general no longer sufficient to guarantee the positive-pair agreement (Lemma 3), and thus further enhances the contrastive consistency objective.

  • Consistently, applying either component alone can NOT outperform GPS+Consistency on all the benchmarks (Table 6), and can NOT effectively mitigate the shortcut solution (Table 7). In contrast, combining both achieves better overall performance and effectively mitigates the shortcut solution (Fig. 3, Fig. 4 and Table 7).

Therefore, only applying BOTH of them can effectively mitigate the shortcut solution, which is consistent with the design of our GCCM.

W1.3 & Q1. Deterministic GPS with the same feature perturbation

We would like to clarify the following points:

  • As mentioned above, the feature perturbation in our GCCM enhances the contrastive consistency objective to mitigate the shortcut solution, and it is NOT designed as a data augmentation. The deterministic GPS does NOT take (Yt,t)(\mathbf{Y}_t, t) as input and has no positive pairs across noise levels, so there is no identical conditioning for the feature perturbation to break.

  • Nevertheless, following your suggestion, we train the deterministic GPS with the same feature perturbation as data augmentation (the same perturbation operator and TperT_{per} as in Table 5), without consistency training, denoted as GPS+Perturbation:

    ModelMNIST (Acc ↑)CIFAR10 (Acc ↑)PATTERN (Acc ↑)CLUSTER (Acc ↑)
    GPS98.08 ± 0.1172.36 ± 0.3286.648 ± 0.06577.780 ± 0.236
    GPS+Perturbation98.10 ± 0.1672.28 ± 0.2186.604 ± 0.03177.935 ± 0.153
    GPS+GCCM (ours)98.24 ± 0.0672.50 ± 0.2986.772 ± 0.04278.820 ± 0.187

    As shown in the table, the same feature perturbation used as data augmentation does NOT substantially boost the performance of the deterministic GPS, and GPS+Perturbation stays below our GCCM on all four benchmarks. We have included this result in Table 2 (Section 5.3) of the revised paper.


W2. Role of the lemmas in our argument

W2.1. Role of Lemma 1

We would like to clarify that the purpose of Lemma 1 is to show that minimizing the vanilla consistency training objective could lead the denoising network to the shortcut solution, and a sufficient solution is exactly what this requires.

As shown in its proof, minimizing this objective requires fθf_\theta to map different inputs to the same output Y\mathbf{Y} for arbitrary (Yt1,t1)(\mathbf{Y}_ {t_1}, t_1) and (Yt2,t2)(\mathbf{Y}_ {t_2}, t_2) (Eq. 19). This requirement can be satisfied by making the input to fθf_\theta independent of (Yt,t)(\mathbf{Y}_ t, t) (Eq. 20), which is achieved by Fsc\mathcal{F}_ {\mathrm{sc}} under the additive fusion in Eq. 3 (Eq. 21). Under Fsc\mathcal{F}_ {\mathrm{sc}}, the training objective reduces to a standard deterministic supervised loss (Eq. 22), so the boundary condition objective can NOT prevent the model from taking the shortcut solution.

Consistently, vanilla consistency training (GPS+Consistency) indeed leads the model toward the shortcut solution (Fig. 2(a)(d), Table 3 and Fig. 4).

W2.2. Role of Lemma 2

We would like to clarify that Lemma 2 concerns the sufficiency of Fsc\mathcal{F}_ {\mathrm{sc}}, rather than its suboptimality.

The shortcut solution arises because dsd^s can be trivially satisfied by ignoring (Yt,t)(\mathbf{Y}_ t, t), as it attains its minimum value 0 under every parameter choice within Fsc\mathcal{F}_ {\mathrm{sc}} (lines 234-249). Since Fsc\mathcal{F}_ {\mathrm{sc}} only requires Wy=0\mathbf{W}_ y = \mathbf{0} and Wt=0\mathbf{W}_ t = \mathbf{0} (Eq. 6), and its sufficiency requires every parameter choice within it to attain the minimum, a counterexample within Fsc\mathcal{F}_ {\mathrm{sc}} can be given (Appendix C.2), showing that Fsc\mathcal{F}_ {\mathrm{sc}} is generally no longer sufficient to minimize dcd^c. This follows from the additional separation requirement across different instances imposed by the negative pairs (lines 296-299).

Consistently, the heatmap visualizations (Fig. 2), the effect of utilizing (Yt,t)(\mathbf{Y}_ t, t) on prediction accuracy (Table 3) and the prediction accuracy under injected labels (Fig. 4) all show that the contrastive consistency objective indeed alleviates the shortcut solution compared with vanilla consistency training.

W2.3. Role of Lemma 3

We would like to clarify that Section 4.3 does NOT contradict Lemma 2: although the contrastive consistency objective makes Fsc\mathcal{F}_ {\mathrm{sc}} no longer trivially sufficient to minimize dcd^c by introducing the additional separation requirement imposed by negative pairs, the optimization may still move toward Fsc\mathcal{F}_ {\mathrm{sc}} by down-weighting Wy\mathbf{W}_ y and Wt\mathbf{W}_ t in Eq. 3 during training, to make positive-pair alignment easier.

The underlying reason is that the contrastive consistency objective contains the positive-pair agreement requirement, which Fsc\mathcal{F}_ {\mathrm{sc}} trivially satisfies under identical conditioning (Eq. 10). This is exactly what motivates feature perturbation, which independently perturbs the input node features to break the identical conditioning input across the two time steps t1t_1 and t2t_2. In this way, Fsc\mathcal{F}_ {\mathrm{sc}} can no longer trivially yield identical latent representations, and is in general no longer sufficient to guarantee the positive-pair agreement (Lemma 3).

Consistently, by introducing feature perturbation, our GCCM further mitigates the shortcut solution compared with GPS+Consistency+Contrastive (Fig. 2, Table 3 and Fig. 4).

Therefore, the lemmas identify the shortcut solution in vanilla consistency training and explain how the two components of our GCCM jointly mitigate it.


Q2. Reducing λ2\lambda_2 in PCL

We would like to clarify the following points:

  • The shortcut solution exists for any λ2>0\lambda_2 > 0: Lemma 1 shows that minimizing the vanilla consistency training objective could lead the denoising network to the shortcut solution, which holds as long as the training objective contains the self-consistency objective, regardless of the value of λ2\lambda_2. To further examine this in practice, we train GPS+Consistency on PATTERN with different λ2\lambda_2, where λ2=0\lambda_2 = 0 removes the self-consistency objective, and report the accuracy with and without (Yt,t)(\mathbf{Y}_t, t) at inference as in Table 3 (a single run for each λ2\lambda_2 due to the limited time of the rebuttal period):

    λ2=0\lambda_2=00.10.30.50.70.9GCCM
    w/o (Yt,t)(\mathbf{Y}_t, t)72.43785.91686.45386.13084.10886.45380.065
    w/ (Yt,t)(\mathbf{Y}_t, t)85.54586.73486.67186.75986.69186.74586.772
    Gain+13.108+0.818+0.218+0.629+2.583+0.292+6.707

    As shown in the table, the shortcut solution still exists for all λ2>0\lambda_2 > 0, where the gain from (Yt,t)(\mathbf{Y}_t, t) remains clearly smaller than that of our GCCM.

  • With λ2=0\lambda_2 = 0, the self-consistency objective, which could lead the denoising network to the shortcut solution (Lemma 1), is removed, so the model no longer ignores (Yt,t)(\mathbf{Y}_t, t) and the gain becomes much larger. However, the model then no longer enforces the self-consistency property of consistency models (Section 3.2), and loses the predictive consistency in PCL that propagates prediction accuracy from low-noise to high-noise conditions for one-step inference, so its accuracy drops.

Therefore, simply reducing λ2\lambda_2 does NOT avoid the shortcut solution. To make the model stop ignoring (Yt,t)(\mathbf{Y}_t, t), the self-consistency objective has to be removed altogether (λ2=0\lambda_2 = 0), but the accuracy then drops. In contrast, our GCCM modifies the consistency objective itself, and thus mitigates the shortcut solution without weakening the consistency objective.


Q3. Mechanism of utilizing (Yt,t)(\mathbf{Y}_t, t) and averaging multiple YT\mathbf{Y}_T

Q3.1. Mechanism of utilizing (Yt,t)(\mathbf{Y}_t, t)

We would like to clarify that the benefit of utilizing (Yt,t)(\mathbf{Y}_t, t) arises during training.

  • With the noisy label as an additional input, the prediction is learned from easy to hard across noise levels: at low noise levels, the noisy label is close to the ground-truth label, so the prediction is easy and accurate; as the noise level increases, the prediction becomes harder, up to pure noise, which is exactly the input at inference. Greater reliance on (Yt,t)(\mathbf{Y}_t, t) means that the model indeed utilizes the noisy label in the easier cases, which helps it learn the harder ones, up to the prediction from pure noise YT\mathbf{Y}_T.

  • Consistently, when the noisy label is injected at different noise levels, our GCCM generally achieves the highest prediction accuracy among the model variants at low and intermediate noise levels (Fig. 4), and it achieves the largest gains from retaining (Yt,t)(\mathbf{Y}_t, t) at inference (Table 3). Moreover, with the same backbone, our GCCM, which takes the noisy label as input, outperforms the deterministic predictor GPS without noise on all seven benchmarks (Table 1).

Q3.2. Averaging multiple YT\mathbf{Y}_T

We would like to clarify that averaging multiple YT\mathbf{Y}_T brings improvements for our GCCM, but these improvements are minor.

  • This is mainly because our GCCM obtains the two noisy labels in training by corrupting the ground-truth label with independently sampled noise, and requires the predictions obtained from these two noisy labels to be consistent. Since YT\mathbf{Y}_T at inference is also sampled noise, the predictions from different YT\mathbf{Y}_T are similar, and thus the improvement from averaging them is limited.

  • To examine this, we evaluate the same trained model of GCCM with one-step inference from a single YT\mathbf{Y}_T, and with the predicted label distributions averaged over 5 and 10 samples of YT\mathbf{Y}_T (a single run with the same seed for each dataset due to the limited time of the rebuttal period):

    InferenceMNIST (Acc ↑)CIFAR10 (Acc ↑)PATTERN (Acc ↑)CLUSTER (Acc ↑)
    Single YT\mathbf{Y}_T98.4772.7586.77278.925
    Average over 5 YT\mathbf{Y}_T98.4972.7486.79178.922
    Average over 10 YT\mathbf{Y}_T98.4972.8086.81778.923

    As shown in the table, averaging multiple YT\mathbf{Y}_T leads to nearly the same accuracy as a single YT\mathbf{Y}_T.

Thank you for the detailed response. The GPS+Perturbation baseline addresses my main question. However, I share the concerns raised by the other reviewers regarding the limited performance improvements. Overall, I think the analysis and empirical diagnosis of the shortcut behavior are interesting contributions. However, I am still not fully convinced that mitigating this behavior necessarily leads to meaningful improvements in prediction performance. I will maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.