acceptodds
Under review as a conference paper at ICLR 2027

Input Relational Structure Shapes Training Dynamics in Neural Networks

Abstract

Input representations are known to influence machine learning models, yet their effects are typically evaluated only after training has converged. We ask whether the training trajectory of learned representations carries a signature of the relational structure encoded in the inputs, and whether this signature predicts robustness. Using molecular property prediction, we compare models operating on graph-, sequence-, and vector-based encodings, and track the linear extractability of chemical properties from their representations across training checkpoints. All three encodings support substantial extraction already at initialization, but differ in how they reorganize it: sequence models largely preserve their initial extractability profile (Spearman between initial and peak extractability), whereas graph and vector models reorganize substantially ( and ), each on different property subsets. To determine whether these representational dynamics are related to robustness, we introduce controlled corruption across all input relational structures and repeat training. Although corruption has little effect on overall task performance, it alters representations in an encoding-specific way: disruption is diffuse and strongest for vectors, intermediate for sequences, and selective for graphs, concentrated on properties that depend on the removed components. Finally, the evolution of extractability during uncorrupted training correlates with corruption sensitivity (), whereas geometric summaries of the same trajectories do not, suggesting representation dynamics as a complementary lens on robustness.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Input inductive bias, Interpretability, Robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_buB9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the inductive biases arising through different input representations and how they change during training. They test this for molecular property prediction models by comparing the linear extractability of chemical properties during training - comparing graph, sequence and vector representations of the same underlying molecules. They find that the extractability changes more for graph and vector-based models during training than for sequence models. Additionally, they analyze how extractability is affected when training for the first N epochs on corrupted molecules.

Critical strengths
  • S1: The central motivation to analyze the different inductive biases that different input representations offer is interesting and important. The field of molecular property predicting seems particularly well suited for this. And the comparison of sequence, graph and vector-based representations makes sense.
Critical weaknesses
  • W1: The cental goal of the paper is to isolate the inductive bias arising through the input structure. However, I am not convinced that the empirical anlaysis can achieve this:
    • The compared runs differ not only in the input structure but in the whole model architecture. E.g. for the vector-based representation a MLP is trained whereas for the graph-case they average performance over a GNN and a Graph Transformer. Thus, it is not clear, whether differences come from the input strucutre (as claimed) or just from a completely different model architecture (that comes with its own inductive biases etc.).
    • Additionally, App. C shows that no hyperparameter tuning for the different models is performed, but identical hyperparameters are used across all datasets and all completely different model families!
  • W2: The empirical results are mostly not surprising/ seem not particular interesting:
    • What seperates this work from existing work is that the authors analyze extractability across training checkpoints. However, Fig 1 a-e shows that the trends observed after training (extractability of graphs highest, sequence lowest) is roughly the same as during all training checkpoints. So it is a bit unclear to me where exactly the value of analyzing the extractability across the complete training trajectoris lies?
    • The authors find that the extractability of properties already encoded in the input format remains most stable during training while properties not directly available in the format change more duing training (ll 294). This seems almost obvious.
  • W3: The authors do not demonstrate any practical advantage that their method might offer for training or selecting molecular-property prediction models. It is unclear what practical relevance their findings have:
    • Their analyis is restricted to testing linear extractability of a fixed set of properties. They do not show the practical relevance of these properties for the tasks we actually care about. E.g., a greater ood generalization of models that have higher extractability.
    • They desing an artifical corruption experiment where the model receives only corrupted data during the first N training epochs. This task is supposed to test "robustness". It is not clear to me how this relates to any practical relevant setting in molecular property prediction.
    • The tested models are all very small: Just 3 layers and 64 hidden dimensions.
Questions for authors
  1. How can you differentiate between the effect of the input structure and other effects arising through e.g. different model architectures or not well tuned hyperparameters?
  2. Where do you see the concrete practical value of probing throughout training vs just at the end?
  3. Does your method offer any concrete/demonstratable practical advantage for training or selcting molecular property prediction models?
  4. I don't fully understand what the corruption experiment is supposed to test/show? How does it relate to any practically relevant scenario?
4 comments

Thank you for taking the time to review our work and for providing detailed feedback. In the following, we provide new analyses that address the questions you raised, focusing on the separation of input structure from architecture and hyperparameters, as well as on the significance and scope of our work.

Q1: We separate the effect of input structure from that of architecture primarily by learning each input encoding with two architectures, and we have now added a residual MLP as a second vector architecture (see our replies to Reviewer 72mk, Q1, and Reviewer HwJv, Q3). In the revised manuscript we report all findings per architecture, and we attribute an effect to the input structure only if it holds for both architectures of an encoding. We consider this criterion reasonable because the two architectures of each encoding process their input differently, by summation and by attention over bonded atoms for graphs, by attention and by recurrence over tokens for sequences, and with and without a skip connection for fingerprints, so an effect that both show is unlikely to arise from either mechanism. Under this criterion, the selectivity of divergence after corruption is positive only for GIN and the Graph Transformer (+0.069 and +0.077 at N = 120), although the Sequence Transformer applies the same attention operation as the Graph Transformer to the tokens of the SMILES string instead of bonded atoms.

Regarding hyperparameter tuning, we would like to note that extensive tuning was unnecessary, as our models perform comparably to published results. The table gives the test performance of each architecture, averaged over 21 runs per dataset (7 seeds and 3 featurizations), next to the range that Jiang et al. (2021) report for eight models under the same protocol, random 80/10/10 splits, with the metrics used there. We found no published results for the remaining datasets under this protocol.

Dataset (metric)Published rangeGINGraph TransformerTransformerBiLSTMMLPResidual MLP
BACE (AUROC)0.84 to 0.900.880.870.850.860.870.86
BBBP (AUROC)0.88 to 0.930.900.900.920.900.890.88
HIV (AUROC)0.80 to 0.830.810.790.780.780.800.80
ESOL (RMSE)0.57 to 0.710.700.730.880.780.750.74
FreeSolv (RMSE)0.85 to 1.331.181.411.621.481.451.38
Lipophilicity (RMSE)0.55 to 0.680.620.710.910.780.730.72

All six architectures lie within the published range on BACE and BBBP. On HIV, GIN lies within it and the other architectures within 0.03 of its lower end.On the three regression datasets, GIN lies within the range, the Graph Transformer and the residual MLP lie 0.02 to 0.08 above its upper end (the same holds for 2/3 datasets for the standard MLP), and the sequence models trail further, consistent with reports that models learned from SMILES need larger datasets (Deng et al., 2023). We therefore have no indication that the shared hyperparameters or model size affect our findings. Moreover, compact models made it possible to train thousands of runs and fit approximately 9 million probes, as the comparison across encodings, architectures, featurizations and seeds requires.

Q2: We would like to note that we do not introduce probing as a method throughout training, since it has been used extensively in studies of language models (e.g., Chiang et al., 2020; Liu et al., 2021). We use it as a measurement tool because our question concerns whether early corruption leaves a persistent trace after full inputs are restored, which cannot be answered from final accuracy since it does not differ between corrupted and uncorrupted runs. Probing at 21 checkpoints gives us property-wise trajectories that allow us to determine whether corrupted runs return to the trajectories of uncorrupted runs, and how it depends on the duration of corruption. This reveals that divergence persists after full inputs are restored, grows with the duration of corruption, and concentrates on properties associated with the removed atoms in graph models. The trajectories also allow us to compare corruption durations over equal-length recovery windows, separating the effect of corruption duration from the time available for recovery (see our reply to Reviewer 72mk, Q2). Finally, this analysis motivated us to examine whether the representational differences have consequences for model predictions, which we address in Q3 and Q4.

Q4: The corruption experiment tests whether information that is withheld early in training and restored afterward has lasting consequences for the trained model, which is a common experimental design when studying critical learning periods (Achille et al., 2019). Furthermore, in molecular property prediction, the design is analogous to fragment- and motif-based pretraining (Zhang et al., 2021; Luong & Singh, 2023) and curriculum learning (Wu et al., 2021). Training on Murcko scaffolds before full molecules is a controlled version of such a training procedure, in which the information withheld early on is known exactly. We agree, however, that it does not establish robustness to distribution shift, and we have removed the robustness framing.

Q3: We believe the relevance of our work lies in what it shows about training and about the role of the input encoding, rather than proposing a method for training or selecting models. More specifically, we provide evidence that the information a model receives early in training can change what its final predictions depend on, without changing its accuracy, with the effect growing with the duration of the early training phase. Furthermore, we show that input encodings shape how the effect of early training is reflected in the learned representations, as only graph models show a representational trace that is more pronounced for properties associated with the removed atoms.

In the revised version of the manuscript, we compared models trained on scaffolds for their first 120 epochs and on full molecules for the remaining 80 with models trained on full molecules throughout (more details about the protocol in our reply to Reviewer HwJv, Q1). Across the six architectures, models with the scaffold-first stage change their predictions less when side chains are removed, by 0.62 to 2.17 standard deviations of the uncorrupted runs of the same configuration, and perform better on scaffold-only inputs, by 1.13 to 1.82 standard deviations. Both differences hold in all six architectures and increase with the duration of the scaffold stage. Moreover, they are of similar magnitude when compared with uncorrupted models trained on full molecules for 90 epochs, ten more than the 80 epochs of full-molecule training received by the corrupted models (0.54 to 1.90 and 1.12 to 2.33, respectively), so the effects cannot be attributed to the shorter period of full-molecule training. Performance on held-out molecules, in contrast, does not differ. Early training therefore changes which parts of the molecule the models rely on, without a detectable change in how well they predict on full molecules.

These findings have implications for how we understand the effects of early training, such as when studying critical learning periods, which have traditionally been characterized by lasting losses in predictive performance. Curriculum learning, in turn, has been found to yield only marginal changes in final performance under standard training conditions (Wu et al., 2021). Our results show why performance alone is not sufficient to characterize these effects, as even when an early training intervention leaves final accuracy unchanged, it can have a lasting effect on what a model learns to rely on. More generally, D'Amour et al. (2022) showed that different training pipelines can produce models with equivalent test performance but different behavior, and our experiments identify the information available early in training as another source of such differences.

This is particularly relevant in molecular property prediction, where compounds often share a scaffold and differ in their substituents, as in lead optimization. Our results suggest a simple way to assess whether a training procedure changes how a model uses the information that was restricted during early training, which is to compare its predictions on full molecules with its predictions on the restricted forms of the same molecules. This comparison measures the dependence of the predictions on the restricted information directly and without probes, and it detects changes in what the model relies on that are not visible in its accuracy.

Summary of changes. In the revised version, we

  • report all results per architecture and featurization, including a residual MLP as a second vector architecture;

  • add the comparison with published benchmarks to the appendix;

  • remove the robustness framing and add the analysis of model predictions after the scaffold stage as Section 5.3;

  • add two discussion points, on critical learning periods and on pipelines with early stages on restricted inputs;

  • move the selection analysis to the appendix

Thank you again for your detailed review. We hope our additions address your concerns, and we would be grateful if you would take them into account in your final assessment. We are happy to answer any further questions.

References

Achille, A., Rovere, M., & Soatto, S. (2019). Critical learning periods in deep networks. International Conference on Learning Representations (ICLR).

Chiang, C.-H., Huang, S.-F., & Lee, H.-y. (2020). Pretrained language model embryology: The birth of ALBERT. Conference on Empirical Methods in Natural Language Processing (EMNLP).

D'Amour, A., Heller, K., Moldovan, D., et al. (2022). Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research, 23(226), 1–61.

Deng, J., Yang, Z., Wang, H., Ojima, I., Samaras, D., & Wang, F. (2023). A systematic study of key elements underlying molecular property prediction. Nature Communications, 14, 6395.

Jiang, D., Wu, Z., Hsieh, C.-Y., et al. (2021). Could graph neural networks learn better molecular representation for drug discovery? A comparison study of descriptor-based and graph-based models. Journal of Cheminformatics, 13, 12.

Liu, L. Z., Wang, Y., Kasai, J., Hajishirzi, H., & Smith, N. A. (2021). Probing across time: What does RoBERTa know and when? Findings of the Association for Computational Linguistics: EMNLP 2021.

Luong, K.-D., & Singh, A. (2023). Fragment-based pretraining and finetuning on molecular graphs. Advances in Neural Information Processing Systems (NeurIPS).

Wu, X., Dyer, E., & Neyshabur, B. (2021). When do curricula work? International Conference on Learning Representations (ICLR).

Zhang, Z., Liu, Q., Wang, H., Lu, C., & Lee, C.-K. (2021). Motif-based graph self-supervised learning for molecular property prediction. Advances in Neural Information Processing Systems (NeurIPS).

Thank you for your rebuttal and your clarificatons.

However, (some of) my concerns could not be resolved:

In particular, I still believe that the method cannot reliably isolate effects of the input structure, e.g. Figure 5 shows large differences between different models using the same input representation. So this questions one of the central claims of the original paper. Additionally, even if two models using the same input pattern show similar trends we cannot attribute this to the input structure. There could still be other explanations such as other shared design choices (e.g. similar aggregation functions).

Additionally, I still believe that hyperparameter tuning is necessary (even if some values are in the range of other published results). Otherwise, it is not possible to isolate effects that might be just due to the fact that the hyperparameters might just better work for some model family compared to another.

And the models trained remain very very small (just 3 layers!). This might overestimate the effect that the input structure plays (becuase the model is too small to learn something by itself).

Lastly, the specific experiment of "probing + Murcko scaffold curricula learning" seems still a bit contrived to me, and it is still difficult to see the relevance of this for practical molecular property prediction.

Because of these reasons I will retain my original score.

Reviewer_72mkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how different molecular input formats relate to changes in neural network representations during training. The authors compare graph inputs, SMILES sequences, and fingerprint vectors across twelve molecular property prediction datasets and five model architectures. At several training checkpoints, they fit linear probes to measure how well the models' representations reveal 32 molecular properties.

The authors find that sequence models largely preserve the ranking of properties by probe score between initialization and each property's peak value. Graph and vector models show larger changes in this ranking. The authors then corrupt the molecular inputs during early training and restore the full inputs afterward. After the switch, the probe trajectories still differ from those of models that always received full inputs. Graph models show more selective effects across properties than vector and sequence models.

The authors define volatility as the average across properties of the standard deviation of probe-score changes between checkpoints during uncorrupted training. For Murcko corruption, volatility correlates with the resulting probe-trajectory divergence (Spearman rho = 0.72). The geometric trajectory measures tested in the paper show much weaker correlations.

Critical strengths

S1. The paper examines changes in learned representations throughout training, not just final model performance.

S2. The paper includes a broad set of experiments and several useful checks.

Critical weaknesses

W1. The experiments do not sufficiently separate the effects of input relational structure from model architecture and featurization.

My main concern is that the paper attributes differences in training dynamics to input structure without fully controlling for model design. The authors use GIN and Graph Transformers for graph inputs, Transformers and BiLSTMs for sequences, and an MLP for vectors. These models differ in their architectures and pooling operations as well as their input formats.

Appendix D.1 lists precomputed structural descriptors, including atomic Randić and Wiener contributions, among the extended graph features. These descriptors may make some topological properties easier to extract with linear probes even before training.

The paper averages results across architectures and featurizations, but these averages do not isolate the effect of input structure. Because the paper centers on that effect, the authors should report the results separately by architecture and featurization, especially for graph models with only basic features. They should also examine whether different pooling methods affect the initial probe scores.

W2. The corruption experiment measures sensitivity to training history rather than robustness under distribution shift.

The authors corrupt molecular inputs during early training and restore the full inputs afterward. They then measure sensitivity using the DTW distance between the post-switch probe trajectories of the corrupted and uncorrupted runs. This distance shows how an earlier intervention affects representation learning. It does not tell us whether the trained model handles new or corrupted inputs better at inference time.

Figure 2 also raises a concern about recovery time because all runs end at epoch 200. Models that switch back to full inputs at epoch 120 have only 80 epochs to recover, compared with 195 epochs for models that switch at epoch 5. The length normalization of DTW does not fully address this difference. The authors should compare runs with equal recovery periods and evaluate volatility against an independent robustness measure.

W3. It is unclear how much predictive value volatility adds beyond simple trajectory statistics and architecture choice.

The authors compute both volatility and corruption sensitivity from probe selectivity trajectories. Models with larger changes in probe scores may also have larger distances between trajectories. This relationship alone would not show that volatility captures a separate aspect of robustness. The sibling-seed and control-probe analyses help address concerns about shared noise, but simpler explanations remain possible. The authors should compare volatility with predictors such as the initial probe score and the increase during the first ten epochs.

The model-selection experiment also needs a baseline that controls for architecture choice. The authors report a 51% reduction in corruption sensitivity with volatility-based selection, but the method selects a BiLSTM for every dataset. They should compare it with a fixed BiLSTM baseline to show whether volatility adds value beyond architecture choice.

Questions for authors
  1. Could the authors report the main probe results separately for each architecture and featurization, especially for graph models with only basic atom and bond features? Do different pooling methods affect the probe scores at initialization or the trends during training?

  2. Does volatility predict model performance under inference-time input perturbations or an out-of-distribution split, such as a molecular scaffold split? Does the increase in divergence shown in Figure 2 remain if all corrupted runs receive the same number of full-input recovery epochs?

  3. Could the authors compare volatility with simpler measures, such as the change in probe selectivity during epochs 0–10 and the initial probe score? How much does volatility-based selection improve on a baseline that always selects BiLSTM? Does it still help when all candidate models use the same architecture?

3 comments

Thank you for taking the time to review our work and for providing detailed feedback. In the following, we provide new analyses that address the questions you raised, focusing on the separation of input structure from architecture, featurization and pooling, on the behavior of the trained models at inference and the recovery time after corruption, and on the comparison of volatility with simpler measures.

Q1: We now report main results separately for each architecture and featurization (Appendix C), and we added a residual MLP as a second vector architecture (described in our reply to Reviewer HwJv, Q3). The result most directly tied to input structure is the selectivity of divergence after corruption. This is the difference in log divergence from uncorrupted runs between directly and indirectly affected properties, with divergence measured relative to the divergence between uncorrupted runs with different seeds (see Q2), where positive values indicate that the divergence concentrates on the properties that depend on the removed atoms.

Model, Murcko corruption, N = 120InputSelectivity of divergence [95% CI]
GINgraph+0.069 [+0.027, +0.105]
GIN, atom type onlygraph+0.093 [+0.025, +0.165]
Graph Transformergraph+0.077 [+0.051, +0.105]
Graph Transformer, atom type onlygraph+0.107 [+0.032, +0.186]
TransformerSMILES−0.014 [−0.079, +0.054]
BiLSTMSMILES−0.008 [−0.113, +0.089]
MLPfingerprint−0.033 [−0.066, −0.001]
Residual MLPfingerprint−0.009 [−0.055, +0.035]

Selectivity of divergence is positive in both graph architectures, which aggregate information by sum and by attention, and it does not differ between them (0.008 [−0.037, 0.055]). It is not positive in either sequence architecture, including the Transformer, which shares the attention mechanism of the Graph Transformer, or in either vector architecture. The precomputed Randić and Wiener terms you point to belong to the extended featurization. With atom type as the only node feature, selectivity remains positive in both graph architectures. In both graph architectures, the topological properties are the most coupled to initialization (higher order 0.95 and 0.89, basic 0.82 and 0.75 for GIN and the Graph Transformer), and this remains so with atom type as the only node feature (0.94 and 0.81). How much each model reorganizes overall, by contrast, varies with architecture (ρ between 0.55 and 0.96). We therefore report it per architecture, indicating which properties of the input structure are affected.

On pooling, both graph models aggregate node embeddings by summation, whereas the sequence models average over tokens, so the readout differs together with the input format. A sum over the atoms of a molecule grows with the number of atoms and therefore makes molecular size linearly available, which an average does not. To test how much of the initial probe scores depends on this, we replaced the sum readout of both graph models with the mean readout, on the identical epoch 0 weights. We report probe selectivity at initialization separately for properties whose Spearman correlation with the number of heavy atoms exceeds 0.8 in absolute value within a dataset, which we call size extensive (about a quarter of all properties), and for the remaining properties.

Selectivity at epoch 0All propertiesSize extensiveRemaining
GIN, sum readout0.7420.9790.661
GIN, mean readout0.5880.6110.580
Graph Transformer, sum readout0.7150.9550.632
Graph Transformer, mean readout0.5400.5280.544
Transformer, mean readout0.5940.7950.525
BiLSTM, mean readout0.5660.6860.525
MLP0.6490.8950.564
Residual MLP0.7270.9400.654

With the mean readout, the initial selectivity of the graph models decreases by 0.37 and 0.43 on size-extensive properties and by 0.08 and 0.09 on the remaining properties. The readout therefore affects the initial probe scores mainly on size-extensive properties. With the readouts used in training, all models reach between 0.69 and 0.98 on these properties. On the remaining properties, the graph models extract at least as much as the sequence models, with either readout (sum: 0.661 and 0.632; mean: 0.580 and 0.544, compared to 0.525 for both sequence models). To test whether the size information drives the trends we attribute to input structure, we repeated the analysis of divergence after Murcko corruption without the size-extensive properties. Selectivity of divergence at N = 120 is still positive for both graph models (GIN +0.056 [+0.014, +0.095], Graph Transformer +0.071 [+0.044, +0.099], compared with +0.069 and +0.077 on all properties). It is not positive for the remaining models (Transformer −0.012 [−0.072, +0.050], BiLSTM −0.012 [−0.080, +0.053], MLP −0.037 [−0.081, +0.000], residual MLP −0.010 [−0.039, +0.021]).

Q2: We agree that trajectory divergence measures how an earlier intervention affects representation learning, not how the trained model handles new or corrupted inputs, so in the revised paper we will no longer describe it as robustness. To answer your question, we evaluated all trained models at inference on inputs reduced to their Murcko scaffold and on a scaffold split, for which we partitioned the held out molecules of each dataset by whether their scaffold occurs in the training set. Volatility does not predict performance under either shift. For runs of the same architecture and dataset, it is not associated with the decrease in performance from full to scaffold inputs (ρ = 0.048 [−0.045, 0.146])).

The corruption itself, however, changes how the trained models respond to scaffold inputs. After 120 epochs of Murcko corruption, models perform better on scaffold inputs than uncorrupted runs of the same configuration, and their predictions change less when the side chains are removed. Both effects hold in each of the six architectures, while test performance remains unchanged and the gap between seen and novel scaffolds changes only for the BiLSTM and the residual MLP, by 0.18 to 0.28 standard deviations. We describe the protocol and discuss the implications of these results in our reply to Reviewer HwJv (Q1), and add them to the paper as Section 5.3.

Regarding recovery time, we agree that the length normalization does not fully address the difference, since dividing by the window length inflates the divergence of short windows. We therefore compared all runs over windows of equal length, the first eight checkpoints after each switch. For each window, we divide the divergence of corrupted runs from uncorrupted runs by the divergence among uncorrupted runs over the same window, so that a ratio of 1 indicates no effect of the corruption.

Murcko / generic corruptionN = 5N = 20N = 60N = 120
GIN1.038 / 1.1121.048 / 1.2191.060 / 1.4091.099 / 1.619
Graph Transformer1.018 / 1.0701.041 / 1.1411.060 / 1.2391.082 / 1.360
Transformer1.035 / 1.1001.076 / 1.1921.048 / 1.2261.049 / 1.276
BiLSTM1.036 / 1.0931.075 / 1.1881.135 / 1.2711.137 / 1.380
MLP1.012 / 1.0231.005 / 1.0301.018 / 1.0711.035 / 1.145
Residual MLP0.996 / 0.9970.998 / 1.0081.031 / 1.0701.038 / 1.112

With equal recovery time, the ratio still increases from N = 5 to N = 120 in every architecture under the generic corruption and the Murcko corruption, where the Transformer peaks at N = 20 and its interval at N = 120 includes 1. The effects on predictions also grow from N = 5 to N = 120 when measured at a matched time after the switch. Figure 2 now shows this ratio over windows of equal length.

Q3: As you suggest, we compared volatility with two simpler summaries of the same trajectories, the mean probe selectivity at initialization and its change over the first ten epochs, which we computed both signed and as an absolute value. For each measure, we report its Spearman correlation with corruption sensitivity, and the partial correlation of volatility with corruption sensitivity after controlling for it. Both are computed over the 648 runs with a corrupted counterpart, with 95% confidence intervals from a bootstrap over datasets.

Measureρ with corruption sensitivityPartial ρ of volatility
Initial selectivity−0.07 [−0.20, +0.07]0.74 [0.54, 0.83]
Change in selectivity, epochs 0 to 10+0.31 [+0.03, +0.46]0.70 [0.53, 0.80]
Absolute change, epochs 0 to 10+0.48 [+0.21, +0.62]0.64 [0.48, 0.76]
Volatility0.73 [0.53, 0.83]—

Initial selectivity is not associated with corruption sensitivity, and across architectures the sign of its correlation changes. The early change is associated with corruption sensitivity in every architecture except the Transformer. After controlling for any of the three measures, the partial correlation of volatility remains between 0.64 and 0.74. Together with the sibling seed (0.72) and control probe (0.48) analyses in the paper, this indicates that volatility predicts corruption sensitivity beyond these simpler summaries. Against a baseline that selects a BiLSTM run at random, choosing the BiLSTM run with the lowest volatility reduces corruption sensitivity by a further 41%, in 12 of 12 datasets. Across the other architectures, the reductions range from 5% to 16%. After a Holm correction for the six architectures, only the BiLSTM result is significant (see our reply to Reviewer HwJv, Q2). Since most of the overall reduction comes from the choice of architecture, we moved the selection analysis to the appendix and no longer present it as a contribution.

Summary of changes. In the revised version, we

  • report all main results per architecture and featurization, including a residual MLP as a second vector architecture;

  • add the pooling analysis at initialization to the appendix;

  • remove the robustness framing and add the analysis of model predictions on scaffold inputs and novel scaffolds as Section 5.3;

  • measure divergence in Section 5.2 relative to the divergence between uncorrupted runs with different seeds, over windows of equal length after the switch;

  • add the comparison of volatility with simpler measures to the appendix;

  • move the selection analysis to the appendix.

Thank you again for your detailed review. We hope our additions address your concerns and would be grateful if you could take them into account in your final assessment. We are happy to answer any further questions.

Reviewer_HwJvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether the way representations form during training carries a signature of the input encoding, and whether that signature predicts robustness. The setup exploits the fact that a molecule admits three encodings of the same entity: graphs, SMILES sequences, and fixed-length fingerprint vectors. They probe 32 chemical properties at 21 checkpoints across 12 MoleculeNet and TDC datasets, with two architectures per structured encoding, for roughly 7.5 million probe fits.

Three findings. First, all three encodings already support substantial extraction at initialization, but they differ in what happens next: sequence models barely reorganize (ρ = 0.90 between initial and peak extractability) while graph and vector models reorganize substantially (0.62 and 0.68), each on different property subsets. The properties that stay coupled to initialization are the ones the format encodes directly, so training mostly reorganizes what the input does not already make explicit.

Second, they corrupt inputs by showing only the Bemis-Murcko scaffold for the first N epochs and then restoring full molecules. Task performance is unaffected (Cohen's d = -0.04, no dose-response), but representations diverge, and the divergence roughly doubles going from N = 5 to N = 120. The shape of that divergence is encoding-specific: selective for graphs, concentrated on properties that depend on the removed atoms, diffuse for vectors, and limited but non-specific for sequences.

Third, they define volatility as the standard deviation of consecutive-checkpoint changes in probe selectivity, computed on uncorrupted runs alone. It correlates with corruption sensitivity at ρ = 0.72, while geometric summaries of the same trajectories (CKA, nearest-neighbour preservation, effective rank) correlate at essentially zero and flip sign across families. Selecting the lowest-volatility run per dataset cuts corruption sensitivity by 51% against random selection, where selecting by validation performance gives nothing. The authors note that in all twelve datasets the selected run was a BiLSTM, so most of that gain comes through architecture choice, with within-architecture reductions of 5 to 41%.

Critical strengths

The dissociation between task performance and representational content is clean, well powered, and the controls are the right ones.

Corrupted runs are statistically indistinguishable from their matched baselines at the task level (Cohen's d = -0.04, CI spanning zero, no dose-response with corruption duration), yet their representations diverge systematically and the divergence roughly doubles from N = 5 to N = 120. That is a real result: something persists from early training that standard evaluation cannot see, and its structure depends on the input encoding rather than on how much reorganization occurred, since graphs and vectors reorganize to a comparable degree but diverge selectively versus diffusely.

What makes this consequential rather than merely suggestive is the set of controls around it. The same molecule under three encodings isolates input relational structure from model class, with two architectures per structured encoding. Probe selectivity is used throughout rather than raw r², the volatility correlation survives being recomputed from a different seed's runs (0.72 to 0.71) and partialing out control-probe volatility (0.48), and cluster bootstraps over datasets, architectures, and featurizations all exclude zero. The geometric summaries are then run through the identical pipeline and come out at essentially zero with sign flips across families, which is the comparison that turns "volatility predicts something" into "content-level dynamics carry information that geometry does not."

The authors also state plainly that the 51% selection gain is largely architecture choice, since the selected run is a BiLSTM in all twelve datasets. That they report this rather than leaving the headline number unqualified is part of why I trust the rest.

Critical weaknesses
  1. Robustness is never actually measured.

"Corruption sensitivity" is DTW divergence between probe trajectories of a corrupted and an uncorrupted run. It is not accuracy under shift, not OOD generalization, not adversarial robustness. Worse, Section 5.2 establishes that the corruption has no task-level effect at all (d = -0.04, no dose-response). So the headline chain is that volatility predicts internal divergence caused by a perturbation the authors have shown does not matter for performance. The abstract says the signature "predicts robustness" and the title of 5.3 says it predicts corruption sensitivity, but nothing in the paper links either quantity to a failure anyone would notice. Until that link exists, this is a correlation between two internal measurements.

  1. The one actionable deliverable reduces to picking an architecture.

Volatility-based selection cuts corruption sensitivity 51% against random, but the selected run is a BiLSTM in all twelve of twelve datasets, and within architectures the reduction falls to 5 to 41%. The average volatilities in 5.3 already separate sequences (0.042) from graphs and vectors (0.070, 0.065), so the same recommendation follows from three numbers without probing anything per run. The authors disclose this, which is to their credit, but disclosure does not convert it into a selection criterion.

  1. One domain, and the vector family rests on a single architecture.

Everything is molecules, including the replication corruption in Appendix H, while the title claims something about neural networks generally. More specifically, the design's stated purpose is to separate input structure from model class, and it does that with two architectures each for graphs and sequences but only an MLP for vectors. The paper's sharpest claim, that graphs and vectors reorganize comparably yet diverge selectively versus diffusely, therefore compares a two-architecture family against a one-architecture family. Diffuse propagation may be an MLP property rather than a property of unstructured input.

Questions for authors

1.Does volatility predict any externally visible failure? You corrupt by removing everything outside the Bemis-Murcko scaffold, so scaffold-split evaluation is where that removal should bite hardest, yet all 12 datasets use random 80/10/10 splits. If low-volatility runs also degrade less under a scaffold split, or under any shift, the robustness framing is earned. As it stands both sides of the ρ = 0.72 are internal measurements.

2.Within each architecture, does selecting by volatility beat selecting at random? You report a 5 to 41% range but not whether the low end is distinguishable from noise, and the 51% headline is confounded by the BiLSTM being selected 12 times out of 12. Put differently, does per-run volatility add anything over the three family-level averages in 5.3?

3.Can you add a second vector architecture? The graph versus vector contrast, selective against diffuse propagation at comparable reorganization, is the paper's sharpest claim, but vectors are represented by one MLP while the other families have two architectures each. A residual or wider MLP would show whether diffuseness is a property of unstructured input or of that model.

4.Is the ρ = 0.72 correlation driven by between-family variance? You report within-family values of 0.89, 0.86, and 0.67, which is reassuring, but the three families also differ systematically in mean volatility. A partial correlation controlling for family would make the within-family claim cleaner.

3 comments

Thank you for taking the time to review our work and for providing detailed feedback. In the following, we provide new analyses that address the questions you raised, focusing on how representational traces of early corruption relate to model predictions, as well as the single-vector architecture.

Q1: We agree that both sides of the correlation are internal measurements, and that a scaffold split is where the removal of side chains should matter most. Since all datasets use random training/validation/test splits, we performed an additional evaluation in which we partitioned the held-out molecules according to whether their Bemis–Murcko scaffold occurs in the training set. This allows us to compare performance on held-out molecules with scaffolds represented in training against performance on held-out molecules with previously unseen scaffolds. For each uncorrupted run, we then measured the decrease in performance from seen scaffolds to novel ones, as well as the decrease in performance when only the Murcko scaffold of a molecule is provided as input (scaffold inputs), as in the corruption described in the manuscript. Since these comparisons involve subsets that differ in class balance and target spread, we evaluate them with AUROC for classification and Spearman correlation for regression, which are robust to such discrepancies. Results show that runs with low volatility do not degrade less under either shift. For runs of the same architecture and dataset, volatility is not associated with the decrease in performance from full to scaffold inputs (ρ = 0.048 [−0.045, 0.146]), and if anything, runs with higher volatility lose slightly less performance on novel scaffolds (ρ = −0.125 [−0.220, −0.035]). We believe that these results do not support the robustness framing, so we will remove it from the abstract and from the volatility section (now Section 5.4) in the revised version of the manuscript. Motivated by your question, we examined whether the corruption itself has consequences visible outside the representation, using the same scaffold inputs. To this end, for every trained model, we measured its performance on scaffold inputs, and how much its predictions change when the side chains are removed, regardless of whether they are correct; that is, the mean absolute difference between its predictions on the full molecules and on their scaffolds, divided by the standard deviation of its predictions on the full molecules. For each corrupted run, we subtract from each quantity the mean of the seven uncorrupted runs of the same dataset, architecture and featurization, and divide by their standard deviation across seeds, and we average the result over runs, with 95% confidence intervals from a bootstrap over datasets. The reported values thus express how much early corruption changes each quantity relative to the variation across random seeds in the corresponding uncorrupted configuration. Negative values for the change in predictions indicate reduced dependence on the side chains, and positive values for performance on scaffold inputs indicate better performance when only the scaffold is provided.

Model, Murcko corruption, N = 120InputChange in predictions when side chains are removedPerformance on inputs reduced to the scaffold
GINgraph−2.17+1.66
Graph Transformergraph−1.77+1.15
TransformerSMILES−0.62+1.13
BiLSTMSMILES−1.37+1.56
MLPfingerprint−1.57+1.68
Residual MLPfingerprint−1.38+1.82

The effects are consistent across all three encoding families and across their respective architectures. After early corruption, models change their predictions less when the side chains are removed and perform better on scaffold inputs than their uncorrupted counterparts. In contrast, we detect no difference in test performance (Cohen’s d = −0.02), and the performance difference between novel and seen scaffolds changes only for the BiLSTM and the residual MLP, whose corrupted runs lose 0.18 to 0.28 standard deviations more on novel scaffolds after 60 or more epochs of corruption. Thus, early corruption changes the extent to which trained models depend on side chain information, even though these changes are largely not reflected in task-level evaluations.

This result has a direct practical implication, since it shows that an early stage of training can change what a trained model depends on, even as the metrics used to evaluate it remain unchanged. Curricula that order training inputs from simple to complex (Bengio et al., 2009) are typically assessed by their effect on final performance (Wu et al., 2021), and our results indicate that such an assessment can miss what the early stage changed. In molecular property prediction in particular, several pretraining methods emphasize information at the level of fragments or motifs before training on the target task (Zhang et al., 2021; Luong & Singh, 2023). Training on Murcko scaffolds before full molecules can be seen as a controlled version of that stage, in which the information withheld early is known exactly. The result also extends the literature on critical learning periods, in which the effect of an early deficit is typically measured as a lasting loss of task performance (Achille et al., 2019; Kleinman et al., 2023). Here, the early deficit has a lasting effect without any loss of task performance, since it determines what the predictions of the trained model depend on. We plan to add this analysis to the paper as Section 5.3, together with two discussion points on the implications of these findings, for the study of critical learning periods and for the evaluation of models trained with early stages on restricted inputs, which we describe in more detail in our reply to Reviewer buB9.

Q2: Within each architecture, we compared the run with the lowest volatility with the expected random run of the same architecture across the 12 datasets. Selection by volatility reduces corruption sensitivity in 12 of 12 datasets for the BiLSTM (by 41% on average), in 10 of 12 for the MLP (6%), in 9 of 12 for the Transformer (16%), the residual MLP (7%) and the Graph Transformer (6%), and in 8 of 12 for GIN (5%). With one sided exact sign tests and a Holm correction for the six architectures, only the BiLSTM result is significant (adjusted p = 0.0015, against 0.10 for the MLP and 0.29 for the others). The overall reduction of 50% therefore mostly reflects the choice of architecture. In light of these results, we will remove the selection analysis from the main text, and move it to the appendix. We would like to note however that per run volatility does predict corruption sensitivity beyond the family averages, since its correlation within families is higher than the pooled one (details in Q4).

Q3: Following your suggestion, we trained a residual MLP for every dataset, featurization and condition (1116 runs), with the same depth and width as all other models and a skip connection around the second hidden layer. To compare it with the graph architectures, we use the selectivity of divergence from Section 5.2. This is the difference in log divergence from uncorrupted runs between directly and indirectly affected properties, where positive values indicate that the divergence concentrates on the directly affected properties. We also report the coupling between initial and peak extractability from Section 5.1.

Model, Murcko corruption, N = 120InputSelectivity of divergence [95% CI]ρ(init, peak)
GINgraph+0.069 [+0.027, +0.105]0.70
Graph Transformergraph+0.077 [+0.051, +0.105]0.55
MLPfingerprint−0.033 [−0.066, −0.001]0.68
Residual MLPfingerprint−0.009 [−0.055, +0.035]0.84

Like the MLP, the residual MLP does not concentrate the divergence on the properties directly affected. Both graph architectures do, with no difference between them (0.008 [−0.037, 0.055]). At the same time, the residual MLP is coupled to its initialization more strongly than the MLP (0.84 against 0.68). The contrast between graph and vector inputs therefore holds for two vector architectures that reorganize to different degrees, suggesting that the structure of the divergence depends on the input encoding rather than on the magnitude of reorganization.

Q4: Controlling for family, the partial correlation between volatility and corruption sensitivity is ρ = 0.795 [0.643, 0.87], and controlling for architecture it is ρ = 0.791 [0.628, 0.87], both higher than the pooled ρ = 0.73. The correlation is therefore not driven by differences between families. It is stronger within them, which also answers the second part of Q2. Geometric summaries of the same trajectories remain weak (|ρ| ≤ 0.48 within every architecture, with signs that change across architectures), so the contrast between content and geometry is unchanged.

Summary of changes. In the revised version, we

  • remove the robustness framing from the abstract and the volatility section;
  • add the analysis of model predictions under scaffold reduction and on novel scaffolds as Section 5.3, with two corresponding discussion points;
  • move the selection analysis to the appendix;
  • add the residual MLP as a second vector architecture;
  • report the partial correlations controlling for family and architecture;
  • state the molecular scope of the study in the title and abstract.

Thank you again for your detailed review. We hope our additions address your concerns and would be grateful if you could take them into account in your final assessment. We are happy to answer any further questions.

References

Achille, A., Rovere, M., & Soatto, S. (2019). Critical learning periods in deep networks. International Conference on Learning Representations (ICLR).

Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009). Curriculum learning. International Conference on Machine Learning (ICML).

Kleinman, M., Achille, A., & Soatto, S. (2023). Critical learning periods for multisensory integration in deep networks. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Luong, K.-D., & Singh, A. (2023). Fragment-based pretraining and finetuning on molecular graphs. Advances in Neural Information Processing Systems (NeurIPS).

Wu, X., Dyer, E., & Neyshabur, B. (2021). When do curricula work? International Conference on Learning Representations (ICLR).

Zhang, Z., Liu, Q., Wang, H., Lu, C., & Lee, C.-K. (2021). Motif-based graph self-supervised learning for molecular property prediction. Advances in Neural Information Processing Systems (NeurIPS).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.