- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 2(+1)
Read review
Paper summary
The authors propose SCARCE. A framework which operates in two cycles to improve OOD generalization for a VGAE encoder. The first, involves a self-supervised contrastive encoder applying reconstruction loss to separate the invariant representation from any shortcuts in the environment, making the representation fixable before stage two. The second, applies pseudo-environments from the frozen anchor clustering into an adversary which predicts the environment as optimized by a uniform confusion loss that can improve alignment with invariants and reduce spurious correlation to the environment.
Critical strengths
The most significant strength of SCARCE is it's capacity to improve downstream OOD performance for the realistic DrugOOD IC50 and EC50 Assay, Scaffold, and Size shifts under the author's proposed learning objective. I believe that this reflects a clever formulation for label-free learning objectives without accidentally exposing testing leakage. I also appreciate the author's efforts to establish clarity within their writing.
Critical weaknesses
I have several concerns, the largest regarding the soundness of the label-free objective, as described in Q1. Largely, I find the consistency of the label-free objective proposed within Equation 2 to be consistent with the standard labelled invariant learning objectives. However, the author propose SCARCE to function in an unlabelled invariant learning environment which is proposed as a means for not testing standard invariance learning methods such as IRM. This is largely concerning since it makes it difficult to consider that a label-free unsupervised OOD problem is sound, especially given that the DrugOOD dataset provides labels for supervised learning. Additionally, the title for the submitted PDF is different than the title listed within OpenReview.
Questions for authors
- This concern is based on my description in the 'Critical Weaknesses' section and the exactness of the statement in Line 47: "Class labels are the expensive part". Couldn't existing invariant learning methods, like IRM, still work with pseudo-environment labels from MVGRL clustering? Or is it unreasonable to treat the pseudo-environments as potential Y label for IRM or other invariance learning techniques? I ask largely because Assay, Scaffold, and Size shift for EC50 and IC50 are dependent on molecular structure or number of nodes, which when encoded could serve a kNN-derived labels. So, it follows that the label-free learning objective within this study (Equation 2) losses it soundness as a practical OOD objective given the potential viability for kNN-derived labels as a target for IRM training.
- Is Line 5 within Algorithm 1 also Equation 7 or is the 'Compute' function dependent on MVGRL clustering as Equation 4, Lines 158, 454-455, and 478 lead me to assume? This affects my current interpretation of how SCARCE determines the proper separation between S and C since it seems that SCARCE relies on MVGRL without explicitly stating it in the current manuscript.
- How does the controlled contraction for geometry overlap and probe regularization function within the shifts of EC50 and IC50? I understand that both datasets are unlikely to indicate as clean separation within Figure 3, but it could be an interesting diagnostic to determine whether the larger success of the Scaffold and Assay split tests is correlated with cleaner separation for S and C. This would fully convince me that the theory proposed within this work is sound.
Rebuttal Response
Thank you to the authors for their rebuttal, I appreciate the clarification on my second and third questions. I have raised my score to reflect that those concerns are addressed on those specific points. I am pleasantly surprised at the effectiveness of SCARCE to manage the proposed label-free objective and believe that the additional rebuttal experiments can be interpreted for useful insight in applying environment confusion to OOD generalization.
On my first concern, I clarify that I do not believe that the entire label-free setting is unsound. My assertion is grounded in how the presentation of the proposed evaluation objective, as it currently stands, in-context to invariant learning and self-supervised/unsupervised settings is confusing. Equation 2 describes a scenario including Y for minimizing risk between environments, just like invariant learning, but then immediately claims that the objective is label-free. I understand this could be done for multiple reasons, including: for effect, is related to experimental setup (Line 361), and that the remainder of Line 180 follows self-supervised learning paradigms. But, it currently reads strangely to only indirectly mention self-supervised learning within Lines 175-184, especially given that SCARCE is centered around a self-supervised/unsupervised learning paradigm.
From a presentation standpoint, a simple one-line update within Lines 175-184 which is explicit about the self-supervised/unsupervised nature of this work would clarify the intent of Line 180 and make the ruling out of existing invariant learning methods no longer read as awkward to me.
From a practical standpoint, my concern was also largely-grounded in that this work's tested IC50 and EC50 DrugOOD datasets do not inherently suffer from a lack of labels, where previous OOD-generalization techniques with full label-access have demonstrated strong performance [1,2]. It is not necessary for additional benchmarking experiments on the two datasets with CIGA and GALA, but clarification on the benefits that SCARCE provides over models like CIGA and GALA will be of great benefit to demonstrating the value of SCARCE's contribution.
I appreciate the author's consideration and look forward to continued discussion.
[1] Chen, Y., Zhang, Y., Bian, Y., Yang, H., Kaili, M.A., Xie, B., Liu, T., Han, B. and Cheng, J., 2022. Learning causally invariant representations for out-of-distribution generalization on graphs. Advances in Neural Information Processing Systems, 35, pp.22131-22148.
[2] Chen, Y., Bian, Y., Zhou, K., Xie, B., Han, B. and Cheng, J., 2023. Does invariant graph learning via environment augmentation learn invariance?. Advances in Neural Information Processing Systems, 36, pp.71486-71519.