- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies when a previously suppressed shortcut recovers during training, how the GNN's robustness changes. The authors run a series of empirical experiments to analyze this observation. The paper further studies the effects of batch size, optimizer state, training-distribution diversity, and environment partitioning. An additional GOOD/MNIST-superpixel experiment shows a regime in which ERM is already robust and shortcut suppression is harmful.
Critical strengths
-
The paper highlights that robustness observed immediately after shortcut suppression may be temporary, and argues that robustness should also be evaluated after the shortcut is restored.
-
The authors study this with controlled trajectory experiments, varying suppression duration and batch size, using optimizer-state/reset controls, and testing different training mixtures and environment partitions, with an additional external graph task for validation.
Critical weaknesses
- Overall, this paper is like a lab experimental report, not a paper. Almost all of the paper is about the experimental details and the results, with nothing about the problem background, motivation, why this problem is important, or why the authors did such a bunch of experiments to study this problem.
- Especially in the abstract and introduction, the background of this problem and what motivates the authors to study this problem need to be clarified. Why this problem is important and what the gap is.
- For now, it just contains a bunch of experimental studies, so it is difficult to assess the novelty of the paper.
Questions for authors
- Is this problem a natural phenomenon during GNN training, where it is common that suppression shortcuts are restored later during training? Under what situations will this happen and affect robustness?
- The paper needs to be reframed, and the questions mentioned above clarified.
2 comments
Thank you for the detailed response. I believe the motivation and explanations provided in your response should be fully integrated into the manuscript to make it more readable and understandable. The current paper still feels incomplete in its presentation. A complete research paper should clearly establish the problem, explain its importance and motivation, identify the research gap, and then present the proposed approach and contributions. I strongly encourage the authors to restructure the paper accordingly rather than focusing primarily on experimental details and results.
Reply to Reviewer nqcP
We accept the framing criticism and will restructure the paper around motivation and question. We answer both questions.
On framing ("a lab experimental report, not a paper"). The root cause is that this draft is a compressed, audit-oriented rewrite: it prioritized traceability (experiment identifiers, protocol inventory, disclosed confounds) over narrative, and the introduction no longer carries the motivation. The revision restructures the abstract and introduction as follows:
A schematic figure of the three regimes will anchor the story, and the main text will use the full 9-page budget.
Q1 (is this a natural phenomenon? under what situations?). Yes — this regime arises whenever the training stream changes after a suppression phase: (i) data-source or mixture changes during long/continual training; (ii) removal or decay of an augmentation that was masking a cue; (iii) the end of curriculum or reweighting stages; (iv) federated/online settings where a biased client re-enters. Our results give a first-order answer to when it matters and affects robustness: when the shortcut is strong and homogeneous across the training distribution, degradation after return is the norm (36/40 runs on RedHouse, onset varying widely across seeds); when shortcut strength is naturally diverse, plain ERM is already robust and release changes nothing (P4). Architecture dependence is now resolved: GCN/GAT replications (20/20 released runs degrade at H=6,000; onset architecture-modulated) will be reported in the revision.
On Soundness 1. We respectfully ask for specifics so we can respond concretely. The design uses ten independent data seeds with seed-paired bootstrap intervals, full checkpoint forks with matched controls (norm-matched and shock-only branches), fixed-data partition controls, and explicitly non-pooled protocols (App. A inventories every family and its limitations). If a particular analysis is in doubt, we will provide the per-seed records during the discussion period.