acceptodds
Under review as a conference paper at ICLR 2027

When Shortcuts Return: Post-Restoration Robustness in Graph Classification

Abstract

Temporarily suppressing a known shortcut can improve robustness to distribution shift, but an endpoint evaluation does not reveal how that robustness changes once the shortcut returns. We study this on a controlled graph-classification task by permuting node colors during a training prefix and then restoring the original inputs. Permanent permutation is the stronger accuracy baseline ( vs. worst-environment). A trajectory experiment with ten independent data seeds and full checkpoint forks then measures the restoration horizon directly: every released run starts above at restoration, 36 of 40 meet a sustained-degradation criterion within 6,000 restored-input updates, and continuous suppression exceeds the released arms by – anti-environment accuracy (seed-paired intervals excluding zero). Batch size modulates re-acquisition at matched update count; a longer prefix helps only at the larger batch. Norm-matched optimizer forks show the reset damage is largely a scale-shock artifact. Diverse training mixtures support robust ERM without suppression, environment grouping changes the effect of an invariance penalty, and an external task with naturally diverse shortcut proportions reverses the ordering—bounding when suppression is the right tool. We distinguish these observations from unresolved mechanisms and give conditional analysis for low CE gradients and fixed-feature replacement.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
shortcut learning, spurious correlation, graph neural networks, out-of-distribution generalization, training dynamics, warmup intervention, gradient dominance, phase transition

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nqcPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies when a previously suppressed shortcut recovers during training, how the GNN's robustness changes. The authors run a series of empirical experiments to analyze this observation. The paper further studies the effects of batch size, optimizer state, training-distribution diversity, and environment partitioning. An additional GOOD/MNIST-superpixel experiment shows a regime in which ERM is already robust and shortcut suppression is harmful.

Critical strengths
  1. The paper highlights that robustness observed immediately after shortcut suppression may be temporary, and argues that robustness should also be evaluated after the shortcut is restored.

  2. The authors study this with controlled trajectory experiments, varying suppression duration and batch size, using optimizer-state/reset controls, and testing different training mixtures and environment partitions, with an additional external graph task for validation.

Critical weaknesses
  1. Overall, this paper is like a lab experimental report, not a paper. Almost all of the paper is about the experimental details and the results, with nothing about the problem background, motivation, why this problem is important, or why the authors did such a bunch of experiments to study this problem.
  2. Especially in the abstract and introduction, the background of this problem and what motivates the authors to study this problem need to be clarified. Why this problem is important and what the gap is.
  3. For now, it just contains a bunch of experimental studies, so it is difficult to assess the novelty of the paper.
Questions for authors
  1. Is this problem a natural phenomenon during GNN training, where it is common that suppression shortcuts are restored later during training? Under what situations will this happen and affect robustness?
  2. The paper needs to be reframed, and the questions mentioned above clarified.
2 comments
AuthorsOpenReview ↗

Reply to Reviewer nqcP

We accept the framing criticism and will restructure the paper around motivation and question. We answer both questions.

On framing ("a lab experimental report, not a paper"). The root cause is that this draft is a compressed, audit-oriented rewrite: it prioritized traceability (experiment identifiers, protocol inventory, disclosed confounds) over narrative, and the introduction no longer carries the motivation. The revision restructures the abstract and introduction as follows:

  • ¶1 Motivation. Suppressing a known shortcut (reweighting, augmentation, masking nuisance input columns) is a standard mitigation. But in realistic continuing training, the suppressed cue frequently re-enters the stream: data sources change, an augmentation that was masking a cue is removed or decays, curriculum/reweighting stages end, and in federated or online settings a biased distribution re-enters. Suppression is often temporary in practice, not a one-shot fix.
  • ¶2 Gap. An endpoint evaluation cannot distinguish durable robustness from transient dependence on training history. No prior work characterizes the post-restoration trajectory of robustness with independent-seed replication and matched controls; "retraining on biased data relearns bias" is a folk expectation, not a measured regularity with known modulators.
  • ¶3 Contributions. (i) Continuous suppression, finite suppression, and post-restoration continuation treated as distinct objects with distinct clocks; (ii) horizon dependence: all released runs start >0.99>0.99 at restoration and 36/40 degrade within 6,000 restored updates (continuous −- released =+0.55/+0.76= +0.55/+0.76, seed-paired 95% intervals excluding zero), with batch size and prefix length as measured modulators; (iii) a boundary condition: with naturally diverse shortcut proportions, plain ERM is already robust and continuous suppression is counterproductive (0.868 vs 0.258); (iv) matched-fork controls separating update-scale shock from optimizer-state removal.

A schematic figure of the three regimes will anchor the story, and the main text will use the full 9-page budget.

Q1 (is this a natural phenomenon? under what situations?). Yes — this regime arises whenever the training stream changes after a suppression phase: (i) data-source or mixture changes during long/continual training; (ii) removal or decay of an augmentation that was masking a cue; (iii) the end of curriculum or reweighting stages; (iv) federated/online settings where a biased client re-enters. Our results give a first-order answer to when it matters and affects robustness: when the shortcut is strong and homogeneous across the training distribution, degradation after return is the norm (36/40 runs on RedHouse, onset varying widely across seeds); when shortcut strength is naturally diverse, plain ERM is already robust and release changes nothing (P4). Architecture dependence is now resolved: GCN/GAT replications (20/20 released runs degrade at H=6,000H{=}6{,}000; onset architecture-modulated) will be reported in the revision.

On Soundness 1. We respectfully ask for specifics so we can respond concretely. The design uses ten independent data seeds with seed-paired bootstrap intervals, full checkpoint forks with matched controls (norm-matched and shock-only branches), fixed-data partition controls, and explicitly non-pooled protocols (App. A inventories every family and its limitations). If a particular analysis is in doubt, we will provide the per-seed records during the discussion period.

Thank you for the detailed response. I believe the motivation and explanations provided in your response should be fully integrated into the manuscript to make it more readable and understandable. The current paper still feels incomplete in its presentation. A complete research paper should clearly establish the problem, explain its importance and motivation, identify the research gap, and then present the proposed approach and contributions. I strongly encourage the authors to restructure the paper accordingly rather than focusing primarily on experimental details and results.

Reviewer_iJTdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how robustness changes when a previously suppressed color shortcut is reintroduced during graph classification training. Through controlled experiments on the synthetic RedHouse benchmark, it shows that keeping the shortcut permanently suppressed gives better worst-environment accuracy, whereas restoring it leads to gradual robustness degradation across later updates. The work systematically analyzes how batch size, length of suppression prefix, and optimizer state resets affect restoration trajectories, attributing most harm from resets to scale-shock artifacts. It further validates on an external task that suppression can be harmful when shortcut strength naturally varies, thereby delineating boundary conditions for suppression-based robustness.

Critical strengths
  1. It tackles an important and underexamined question: rather than evaluating shortcut suppression only at the endpoint, it characterizes the full trajectory of robustness after the shortcut reappears and introduces the notion of a restoration horizon.
  2. The experimental setup is rigorous and systematic: it employs ten independent data seeds, full checkpoint forks, and matched controls (such as norm-matched optimizer forks) to disentangle confounders, including update scale.
  3. It clearly separates three regimes—continuous suppression, finite suppression, and post-restoration training—and empirically shows that the robustness improvements from finite suppression do not persist permanently.
Critical weaknesses
  1. The core RedHouse task depends heavily on structural statistics (e.g., degree histograms) for label prediction, and the paper does not show that message passing is required for the observed shortcut competition, which limits its relevance to more complex graph reasoning tasks.
  2. The environment partition experiment (V-REx) confounds shortcut reliability with class composition and structural statistics across topology groups, so it cannot cleanly isolate the effect of shortcut reliability by itself.
  3. The supporting theoretical analysis relies on simplified linear and full-batch assumptions that do not fully capture the nonlinear, stochastic training dynamics of the GNN experiments, so it offers only loose intuition rather than a mechanistic explanation.
  4. Most experiments use synthetic datasets; real-world graph benchmarks are analyzed only superficially and do not validate the restoration effect, raising concerns about external validity.
Questions for authors

Please refer to the weaknesses.

Reviewer_qSzSOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The author constructed a graph classification task, in which the ground truth labels are determined by the structure of the graph. However, due to the high correlation between node color and these labels, the model tends to rely solely on color. In the initial stage of training, colors are scrambled to force the model to learn other more relevant information. Subsequently, the original color was restored to observe whether the robustness of the model would continue. The author demonstrates that although the model may perform well in the short term after restoring "shortcuts" (color information), this does not guarantee long-term robustness. In an experiment covering 40 different training trajectories - derived from 10 independent data seeds and 4 different 'color restoration' settings -36 trajectories exhibited sustained performance degradation within 6000 steps of continuous training.

Critical strengths
  1. The author's advancement of evaluation from the endpoint to the training trajectory has certain research value.
  2. This article reports multiple independent data seeds, inheriting optimizer states and data streams, and reporting seed pairing intervals.
  3. This article honestly reports that permanent disruption of known shortcuts is superior to temporary disruption. And it retains the reverse results on external tasks. Continuous inhibition in the GOOD task is actually harmful, without only selecting favorable results for display.
Critical weaknesses
  1. This article does not seem like a mature complete draft, the main text should be written on 9 pages.
  2. The core P2 results are mainly squeezed into a large paragraph on page 5, and it is recommended to list them separately in a table or pair them with a difference chart.
  3. The core phenomenon of this question is valid, but there is still a lack of innovative new conclusions.
  4. Has the result of the main task not been extended to tasks that truly rely on graph structure inference?
  5. The cross entropy gradient boundary requires Jacobian and margin conditions that have not been validated in experiments. Do these propositions provide predictions that can be tested in the main experiment?
Questions for authors
  1. Compared to 'continuing to train on biased data will relearn bias', what predictable and verifiable patterns does this article further reveal? Can a certain amount before training or during release be used to predict the time of degradation occurrence?
  2. After switching to different GNN architectures, is the degradation pattern still consistent after release?
  3. Since there is still severe degradation after norm matching, why can the reset damage be mainly attributed to scale shock?
  4. How to rule out that performance degradation mainly comes from suppressing operational damage and effective input, rather than environmental diversity?
1 comment
Reviewer_qSzSOpenReview ↗

Guidelines on the Use of AI Tools for Manuscript Review

This review process utilized AI to translate and organize the core content of the manuscript. Subsequently, human reviewers conducted a thorough assessment and finalized the review results. Human oversight ensured the overall quality and formatting of the manuscript, and the final conclusions were determined by human judgment rather than by AI.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.