- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In out-of-distribution settings, existing GNN methods typically exploit the extraction of causal subgraphs in order to improve generalization capabilities which ignores the cross-sample generalization cues provided by the spurious subgraphs (complement of the causal subgraph). In this paper, the authors focus on concept shift and find that spurious subgraphs can provide predictive cues that conflict with the predictive cues provided by the causal subgraphs (termed cue discordance). The authors find that when this phenomenon occurs, predictive accuracy suffers. Specifically, the ratio of discordant samples set scales roughly linearly with OOD accuracy. The authors then prove that, on discordant samples, a classifier that only sees the causal subgraph must have strictly lower prediction loss than a classifier that only sees the spurious subgraph by some margin (D3C criterion). Based on this theory, the authors propose the DCS method which includes a learnable edge mask that splits each input graph into a candidate causal subgraph and a candidate spurious subgraph and a shared predictor that outputs predictions for the candidate causal subgraph, the candidate spurious subgraph, and the entire input graph. Then, the model is trained via a three component loss function: CE on candidate causal subgraph and full graph, hinge term that is high when causal subgraph CE isn't lower than spurious subgraph CE, and a compactness regularization term. The final prediction is done on the extracted candidate subgraph. The method is evaluated on the GOOD benchmark (concept shift) and achieves SOTA results.
Critical strengths
- The cue discordance analysis on OOD generalization for graphs is novel and interesting. It provides a fresh viewpoint on why models misclassify OOD samples and how causal subgraphs should be extracted. It is also supported by strong empirical findings that shows predictive accuracy scales roughly linearly with the ratio of discordant samples.
Critical weaknesses
- The existence of discordant samples in the training set is not supported. The paper provides evidence that there are significant amounts of discordant samples in the test set of Motif-basis and CMNIST-color (Figure 2). However, their theory and proposed method depends on the existence and prevalence of discordant samples in the training set. Specifically, if there are very few (or even zero) discordant samples in the training set, their novel relative-risk loss term will be almost constant across all training samples. Then, the training loss would degenerate to only the task loss term and the compactness regularization term. For example, in the default Motif-basis dataset, ~97% of the training samples are forced to have a spurious subgraph that is correlated with the label which means there are at most 3% (practically ~2%) discordant samples in the training set. In this case, the relative-risk term of the proposed loss function (second term) would contribute very little to model training.
- There are inconsistencies with the reported results in the main results table (Table 1). The reported results for some of the baselines are inconsistent with their original papers. Most notably, for PrunE, the original paper reports 79.50±1.57 and 90.28±1.72 on HIV-size and Motif-basis (Yao et al, 2025) which would both be significantly higher than the proposed method; while this paper reports 69.73(0.79) and 77.63(1.74) for PrunE respectively. I haven't checked every baseline but IRM, ERM, and iMoLD results also seem to be somewhat inconsistent with their original papers (Gui et al, 2022; Zhuang et al, 2023).
Minor:
- The CMNIST-color dataset contains 10 digits and 10 colors (Gui et al, 2022) which means the test set should have ~90% discordant samples yet Figure 2 reports ~67%.
Questions for authors
The questions correspond to the weaknesses.
- Could the authors provide evidence that demonstrates the existence/prevalence of discordant samples in the training set of the real datasets? Furthermore, could the authors provide some additional analysis on the scale and contribution of each loss term?
- Could the authors account for the discrepancies between the baseline results in Table 1 and the results reported in the corresponding original papers?
7 comments
W1/Q1. Thank you for clarifying the motivation of the method and how the relative-risk term acts on all training samples. I consider this point addressed.
W2/Q2. The original discrepancies between the reported results and Table 1 of the paper (PrunE: 90.28 -> 77.63 on Motif-basis and 79.50 -> 69.73 on HIV-size, iMoLD: 77.43 -> 67.87 on HIV-size) are still not explained. Randomness in data splitting or graph generation does not explain gaps of this size, given that your reported standard deviations are 1 to 2 points. Could the authors explain what accounts for these differences?
Furthermore, Sec 5.4 claims the method is stable over a broad range of hyperparameter values (λ_rel and δ), yet the rebuttal reports significant increases from 84.86 to 92.14, 72.56 to 76.54, and 74.47 to 81.33 on Motif-basis, HIV-scaffold and HIV-size from a more extensive hyperparameter search. Was the new search over λ_rel and δ? If so, how is this consistent with the stability claim? If not, then which hyperparameters did it cover? The GOOD benchmark fixes the general training settings (e.g., layer count and hidden dimension) and tunes only method-specific hyperparameters. If the new search departs from that without re-running the baselines under the same protocol, the comparison is not fair. Could the authors provide the exact tuning protocol for the proposed method and the evaluated baselines?
We thank the reviewer for raising these important concerns. We carefully provide two major sources of discrepancies between our reported results and those in the original papers.
(1) Motif dataset: Differences in dataset generation.
PrunE uses the pre-generated Motif dataset downloaded from the official GOOD repository, whereas our original experiments regenerated the dataset using the original GOOD Benchmark code. We verified that our generation code and configurations were consistent with those provided by GOOD. Importantly, all results reported in our original paper were obtained using this same regenerated dataset, ensuring consistent experimental conditions across methods. In our additional experiments, we adopted the officially released GOOD dataset, obtaining an accuracy of 92.14% on Motif-basis. We also observed improvements for other baselines, including ERM (78.66 → 81.31 and 67.77 → 70.83) and IRM (78.67 → 80.37 and 67.11 → 70.15).
(2) HIV dataset: Differences in molecular feature processing.
We further identified important differences between the original implementations of PrunE/iMoLD and the standard GOOD Benchmark pipeline:
- Molecular encoders: Both PrunE (
gcn.py) and iMoLD (models/gnnconv.py) employ different specialized atom encoders that are not included in the standard GOOD implementation. These encoders explicitly process molecular atom and bond attributes, whereas our original implementation (for all methods) directly projected the input features into a 300-dimensional hidden space without such specialized molecular encoding. - Additional handcrafted features: iMoLD further extracts 38 handcrafted molecular features, introducing additional feature engineering beyond the standard GOOD pipeline.
Since these preprocessing and encoding strategies are not part of the standard GOOD implementation and differ across methods, our original experiments adopted a unified feature-processing pipeline rather than incorporating these method-specific modifications, with the aim of maintaining consistent input processing across methods.
We appreciate the reviewer for highlighting the importance of explicitly distinguishing benchmark-standardized comparisons from faithful reproductions of the original methods.
We thank for your valuable comments.
We thank the reviewer for raising this concern. We clarify that our method is motivated by the generalizability of non-causal subgraphs, rather than being designed for a particular distribution of discordant training samples. Since non-causal subgraphs can generalize across samples, they may induce classifiers to rely on spurious predictive cues instead of causal mechanisms, even when these cues largely agree with the ground-truth labels during training.
Importantly, cue discordance can persist even under strong spurious correlations, as explaining the labels perfectly is impossible for non-causal cues. Although such conflicts may be rare during training, distribution shifts can substantially amplify them at test time, exposing the limitations of spurious reliance. Cue discordance thus reveals the competition between causal and non-causal predictive mechanisms, rather than serving as a prerequisite for our optimization.
Specifically, Eq.17 contrasts the predictive losses of selected and complementary subgraphs on all training samples, with discordance-aware weighting only adjusting their relative importance. Even if the discordance-aware weights become uniform, the objective reduces to an unweighted relative-risk contrast rather than constant.
Together with the ablation above, these results confirm relative-risk discrimination provides meaningful optimization signals beyond discordant samples, enabling the model to address competing spurious cues during training before their adverse effects become more pronounced under distribution shifts.
For ERM and IRM, we used the official GOOD implementations, obtaining results consistent with those originally reported in the GraphOOD benchmark. For PrunE and iMoLD, we adopted the official implementations and settings provided by thepaper. Furthermore, we conducted a more extensive hyperparameter search for our method, achieving better performance than their originally reported results:
Moreover, minor discrepancies may arise from randomness in data splitting on real-world datasets, as the concept-shift setting involves stochastic partitioning based on domain–label correlations. For synthetic datasets, the random graph generation process may also introduce additional variability beyond data splitting.
Importantly, spurious correlations in real-world datasets are statistical rather than deterministic. For example, in the SST2-length training set, negative samples are substantially longer than positive ones on average (15.74 vs. 5.05). Nevertheless, 25.53% of negative samples fall below the overall average length, while 7.89% of positive samples exceed it. This demonstrates that spurious length cues do not consistently align with ground-truth labels, providing evidence of cue discordance within the training distribution:
Moreover, real-world cue discordance can arise from multiple interacting spurious factors and cannot be fully characterized by a binary partition based on a single attribute. This further motivates our soft weighting strategy, which uses full-graph classification loss as an observable proxy to adaptively emphasize samples with potentially stronger cue conflicts, rather than requiring explicit identification of discordant samples.
Regarding the contribution and scale of each loss term, we present the ablation results and the loss magnitudes at different training stages on Motif-basis. These analyses help clarify how the different objectives jointly influence optimization beyond their effects on final predictive performance: