- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies the inductive biases arising through different input representations and how they change during training. They test this for molecular property prediction models by comparing the linear extractability of chemical properties during training - comparing graph, sequence and vector representations of the same underlying molecules. They find that the extractability changes more for graph and vector-based models during training than for sequence models. Additionally, they analyze how extractability is affected when training for the first N epochs on corrupted molecules.
Critical strengths
- S1: The central motivation to analyze the different inductive biases that different input representations offer is interesting and important. The field of molecular property predicting seems particularly well suited for this. And the comparison of sequence, graph and vector-based representations makes sense.
Critical weaknesses
- W1: The cental goal of the paper is to isolate the inductive bias arising through the input structure. However, I am not convinced that the empirical anlaysis can achieve this:
- The compared runs differ not only in the input structure but in the whole model architecture. E.g. for the vector-based representation a MLP is trained whereas for the graph-case they average performance over a GNN and a Graph Transformer. Thus, it is not clear, whether differences come from the input strucutre (as claimed) or just from a completely different model architecture (that comes with its own inductive biases etc.).
- Additionally, App. C shows that no hyperparameter tuning for the different models is performed, but identical hyperparameters are used across all datasets and all completely different model families!
- W2: The empirical results are mostly not surprising/ seem not particular interesting:
- What seperates this work from existing work is that the authors analyze extractability across training checkpoints. However, Fig 1 a-e shows that the trends observed after training (extractability of graphs highest, sequence lowest) is roughly the same as during all training checkpoints. So it is a bit unclear to me where exactly the value of analyzing the extractability across the complete training trajectoris lies?
- The authors find that the extractability of properties already encoded in the input format remains most stable during training while properties not directly available in the format change more duing training (ll 294). This seems almost obvious.
- W3: The authors do not demonstrate any practical advantage that their method might offer for training or selecting molecular-property prediction models. It is unclear what practical relevance their findings have:
- Their analyis is restricted to testing linear extractability of a fixed set of properties. They do not show the practical relevance of these properties for the tasks we actually care about. E.g., a greater ood generalization of models that have higher extractability.
- They desing an artifical corruption experiment where the model receives only corrupted data during the first N training epochs. This task is supposed to test "robustness". It is not clear to me how this relates to any practical relevant setting in molecular property prediction.
- The tested models are all very small: Just 3 layers and 64 hidden dimensions.
Questions for authors
- How can you differentiate between the effect of the input structure and other effects arising through e.g. different model architectures or not well tuned hyperparameters?
- Where do you see the concrete practical value of probing throughout training vs just at the end?
- Does your method offer any concrete/demonstratable practical advantage for training or selcting molecular property prediction models?
- I don't fully understand what the corruption experiment is supposed to test/show? How does it relate to any practically relevant scenario?
4 comments
Q4: The corruption experiment tests whether information that is withheld early in training and restored afterward has lasting consequences for the trained model, which is a common experimental design when studying critical learning periods (Achille et al., 2019). Furthermore, in molecular property prediction, the design is analogous to fragment- and motif-based pretraining (Zhang et al., 2021; Luong & Singh, 2023) and curriculum learning (Wu et al., 2021). Training on Murcko scaffolds before full molecules is a controlled version of such a training procedure, in which the information withheld early on is known exactly. We agree, however, that it does not establish robustness to distribution shift, and we have removed the robustness framing.
Q3: We believe the relevance of our work lies in what it shows about training and about the role of the input encoding, rather than proposing a method for training or selecting models. More specifically, we provide evidence that the information a model receives early in training can change what its final predictions depend on, without changing its accuracy, with the effect growing with the duration of the early training phase. Furthermore, we show that input encodings shape how the effect of early training is reflected in the learned representations, as only graph models show a representational trace that is more pronounced for properties associated with the removed atoms.
In the revised version of the manuscript, we compared models trained on scaffolds for their first 120 epochs and on full molecules for the remaining 80 with models trained on full molecules throughout (more details about the protocol in our reply to Reviewer HwJv, Q1). Across the six architectures, models with the scaffold-first stage change their predictions less when side chains are removed, by 0.62 to 2.17 standard deviations of the uncorrupted runs of the same configuration, and perform better on scaffold-only inputs, by 1.13 to 1.82 standard deviations. Both differences hold in all six architectures and increase with the duration of the scaffold stage. Moreover, they are of similar magnitude when compared with uncorrupted models trained on full molecules for 90 epochs, ten more than the 80 epochs of full-molecule training received by the corrupted models (0.54 to 1.90 and 1.12 to 2.33, respectively), so the effects cannot be attributed to the shorter period of full-molecule training. Performance on held-out molecules, in contrast, does not differ. Early training therefore changes which parts of the molecule the models rely on, without a detectable change in how well they predict on full molecules.
These findings have implications for how we understand the effects of early training, such as when studying critical learning periods, which have traditionally been characterized by lasting losses in predictive performance. Curriculum learning, in turn, has been found to yield only marginal changes in final performance under standard training conditions (Wu et al., 2021). Our results show why performance alone is not sufficient to characterize these effects, as even when an early training intervention leaves final accuracy unchanged, it can have a lasting effect on what a model learns to rely on. More generally, D'Amour et al. (2022) showed that different training pipelines can produce models with equivalent test performance but different behavior, and our experiments identify the information available early in training as another source of such differences.
This is particularly relevant in molecular property prediction, where compounds often share a scaffold and differ in their substituents, as in lead optimization. Our results suggest a simple way to assess whether a training procedure changes how a model uses the information that was restricted during early training, which is to compare its predictions on full molecules with its predictions on the restricted forms of the same molecules. This comparison measures the dependence of the predictions on the restricted information directly and without probes, and it detects changes in what the model relies on that are not visible in its accuracy.
Summary of changes. In the revised version, we
-
report all results per architecture and featurization, including a residual MLP as a second vector architecture;
-
add the comparison with published benchmarks to the appendix;
-
remove the robustness framing and add the analysis of model predictions after the scaffold stage as Section 5.3;
-
add two discussion points, on critical learning periods and on pipelines with early stages on restricted inputs;
-
move the selection analysis to the appendix
Thank you again for your detailed review. We hope our additions address your concerns, and we would be grateful if you would take them into account in your final assessment. We are happy to answer any further questions.
References
Achille, A., Rovere, M., & Soatto, S. (2019). Critical learning periods in deep networks. International Conference on Learning Representations (ICLR).
Chiang, C.-H., Huang, S.-F., & Lee, H.-y. (2020). Pretrained language model embryology: The birth of ALBERT. Conference on Empirical Methods in Natural Language Processing (EMNLP).
D'Amour, A., Heller, K., Moldovan, D., et al. (2022). Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research, 23(226), 1–61.
Deng, J., Yang, Z., Wang, H., Ojima, I., Samaras, D., & Wang, F. (2023). A systematic study of key elements underlying molecular property prediction. Nature Communications, 14, 6395.
Jiang, D., Wu, Z., Hsieh, C.-Y., et al. (2021). Could graph neural networks learn better molecular representation for drug discovery? A comparison study of descriptor-based and graph-based models. Journal of Cheminformatics, 13, 12.
Liu, L. Z., Wang, Y., Kasai, J., Hajishirzi, H., & Smith, N. A. (2021). Probing across time: What does RoBERTa know and when? Findings of the Association for Computational Linguistics: EMNLP 2021.
Luong, K.-D., & Singh, A. (2023). Fragment-based pretraining and finetuning on molecular graphs. Advances in Neural Information Processing Systems (NeurIPS).
Wu, X., Dyer, E., & Neyshabur, B. (2021). When do curricula work? International Conference on Learning Representations (ICLR).
Zhang, Z., Liu, Q., Wang, H., Lu, C., & Lee, C.-K. (2021). Motif-based graph self-supervised learning for molecular property prediction. Advances in Neural Information Processing Systems (NeurIPS).
Thank you for your rebuttal and your clarificatons.
However, (some of) my concerns could not be resolved:
In particular, I still believe that the method cannot reliably isolate effects of the input structure, e.g. Figure 5 shows large differences between different models using the same input representation. So this questions one of the central claims of the original paper. Additionally, even if two models using the same input pattern show similar trends we cannot attribute this to the input structure. There could still be other explanations such as other shared design choices (e.g. similar aggregation functions).
Additionally, I still believe that hyperparameter tuning is necessary (even if some values are in the range of other published results). Otherwise, it is not possible to isolate effects that might be just due to the fact that the hyperparameters might just better work for some model family compared to another.
And the models trained remain very very small (just 3 layers!). This might overestimate the effect that the input structure plays (becuase the model is too small to learn something by itself).
Lastly, the specific experiment of "probing + Murcko scaffold curricula learning" seems still a bit contrived to me, and it is still difficult to see the relevance of this for practical molecular property prediction.
Because of these reasons I will retain my original score.
Thank you for taking the time to review our work and for providing detailed feedback. In the following, we provide new analyses that address the questions you raised, focusing on the separation of input structure from architecture and hyperparameters, as well as on the significance and scope of our work.
Q1: We separate the effect of input structure from that of architecture primarily by learning each input encoding with two architectures, and we have now added a residual MLP as a second vector architecture (see our replies to Reviewer 72mk, Q1, and Reviewer HwJv, Q3). In the revised manuscript we report all findings per architecture, and we attribute an effect to the input structure only if it holds for both architectures of an encoding. We consider this criterion reasonable because the two architectures of each encoding process their input differently, by summation and by attention over bonded atoms for graphs, by attention and by recurrence over tokens for sequences, and with and without a skip connection for fingerprints, so an effect that both show is unlikely to arise from either mechanism. Under this criterion, the selectivity of divergence after corruption is positive only for GIN and the Graph Transformer (+0.069 and +0.077 at N = 120), although the Sequence Transformer applies the same attention operation as the Graph Transformer to the tokens of the SMILES string instead of bonded atoms.
Regarding hyperparameter tuning, we would like to note that extensive tuning was unnecessary, as our models perform comparably to published results. The table gives the test performance of each architecture, averaged over 21 runs per dataset (7 seeds and 3 featurizations), next to the range that Jiang et al. (2021) report for eight models under the same protocol, random 80/10/10 splits, with the metrics used there. We found no published results for the remaining datasets under this protocol.
All six architectures lie within the published range on BACE and BBBP. On HIV, GIN lies within it and the other architectures within 0.03 of its lower end.On the three regression datasets, GIN lies within the range, the Graph Transformer and the residual MLP lie 0.02 to 0.08 above its upper end (the same holds for 2/3 datasets for the standard MLP), and the sequence models trail further, consistent with reports that models learned from SMILES need larger datasets (Deng et al., 2023). We therefore have no indication that the shared hyperparameters or model size affect our findings. Moreover, compact models made it possible to train thousands of runs and fit approximately 9 million probes, as the comparison across encodings, architectures, featurizations and seeds requires.
Q2: We would like to note that we do not introduce probing as a method throughout training, since it has been used extensively in studies of language models (e.g., Chiang et al., 2020; Liu et al., 2021). We use it as a measurement tool because our question concerns whether early corruption leaves a persistent trace after full inputs are restored, which cannot be answered from final accuracy since it does not differ between corrupted and uncorrupted runs. Probing at 21 checkpoints gives us property-wise trajectories that allow us to determine whether corrupted runs return to the trajectories of uncorrupted runs, and how it depends on the duration of corruption. This reveals that divergence persists after full inputs are restored, grows with the duration of corruption, and concentrates on properties associated with the removed atoms in graph models. The trajectories also allow us to compare corruption durations over equal-length recovery windows, separating the effect of corruption duration from the time available for recovery (see our reply to Reviewer 72mk, Q2). Finally, this analysis motivated us to examine whether the representational differences have consequences for model predictions, which we address in Q3 and Q4.