acceptodds
Under review as a conference paper at ICLR 2027

Improving the generalization capabilities of equivariant message passing neural networks for binding affinity prediction

Abstract

Structure-based models of protein–ligand binding affinity have long been benchmarked by training on PDBbind and testing on CASF-2016, a train/test split in which nearly half of the test complexes have close analogs in the training data. Many recent models developed on splits that remove these analogs rely on pretrained embeddings. We ask the complementary question of which inductive biases are worth building into an equivariant message passing network (MPNN) trained from scratch, and focus on how the network can use the unbound state of a complex, obtained by removing intermolecular edges, in a way that is advantageous to generalization. We find that, in a basic equivariant MPNN trained on PDBbind CleanSplit, adding the unbound state as a second pooled input does not improve accuracy. We notice that, because the two states contain the same atoms, their embeddings can be combined at each atom, at each intermolecular edge and over the pocket using a variety of operations. We refer to these operations as paired readouts, and perform experiments to find a combination of them that enhances binding affinity prediction. The resulting model, PEAR (Paired Embeddings of Atoms and Residues), is the most accurate of the compared models on CASF-2016, including those with pretrained embeddings, and removing any of its paired readouts lowers its accuracy. On ATOM3D LBA at 30% sequence identity, a variant of PEAR matches the best reported accuracy. On seven held-out pocket clusters, a residue-scale variant has the highest average correlation of any single model after finetuning on 25 complexes of each target, and finetuning improves each of our models on every cluster.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Equivariance, equivariant neural networks, binding affinity, binding affinity prediction, binding affinity generalization, PDBBind, Cleansplit, message passing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_24xoOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PEAR predicts protein–ligand binding affinity using a shared encoder for bound and nominally unbound graphs, followed by paired readouts at atom, edge, and pocket levels. Evaluations on CleanSplit, ATOM3D LBA, and held-out pocket clusters show competitive results without pretraining. However, the unbound-branch definition and attribution of the improvements need clarification.

Critical strengths
  1. The paper addresses a relevant question: how to combine corresponding structural representations before graph-level pooling.
  2. The authors provide comparisons of pairing operations, component ablations, repeated runs, and evaluations across several protocols.
  3. Some configurations are competitive without pretraining, and the paper appropriately notes that the LBA30 improvement over CheapNet is not decisive.
Critical weaknesses
  1. The unbound branch is unclear. Appendix D (p. 16, Eqs. 18–19) describes edge-state updates on intermolecular node pairs and feedback in both branches. This appears to allow protein–ligand information exchange despite removing intermolecular message-passing edges. The authors should clarify the implementation and its interpretation.
  2. The ablations do not fully separate paired readouts from added capacity or richer local features. A bound-only model with the same readout structure and parameter-matched local MLP controls would help. The preliminary graph-level concatenation test also uses a different backbone from PEAR.
  3. Generalization gains vary by setting. PEAR slightly improves CASF-2016 RMSE over GEMS (1.258 vs. 1.274), but both PEAR variants have worse RMSE on the independent subset, and pre-finetuning results on held-out clusters trail GEMS. All three evaluation protocols use PDBbind-derived data.
  4. Uncertainty and model selection need clearer reporting. The paper does not provide paired confidence intervals for small differences, and pocket-cluster finetuning uses one fixed 25-example support set per cluster. The selection process for model variants should also be clarified.
  5. The existing figures classify prior work but do not show PEAR’s pipeline. Please add a method diagram showing both branches, edge feedback, paired readouts, and the prediction head, consistent with the equations and algorithm.
Questions for authors
  1. Does the unbound branch apply intermolecular edge feedback? What changes when all cross-molecule communication is disabled?
  2. Can you provide parameter-matched and bound-only controls, and clarify whether architecture choices were made using validation data only?
3 comments

We thank the reviewer for the detailed comments. We ran several new experiments in response and report them below.

1. On removing intermolecular edges. Adhering to this reporting scheme, we have the following result when intermolecular edges are removed from the bound copy in PEAR, equivalent to running the model on two "unbound" masked copies of the molecule. We find that the lower and higher ends of the 95% paired confidence interval are worse by 0.063 and 0.200 RMSE, respectively:

RMSE95% CIPearson95% CI
PEAR1.258[1.157, 1.356]0.834[0.798, 0.866]
intermolecular edges removed from bound copy1.389[1.270, 1.507]0.776[0.724, 0.821]
paired difference+0.131[+0.063, +0.200]−0.058[−0.091, −0.029]

CASF-2016, 5-fold CV ensemble, 285 complexes, 10,000 bootstrap resamples.

2. Clarifying how persistent edge states and edge feedback affect the unbound copy of the complex. There is no message passing between the protein and the ligand in the unbound copy. After the preliminary experiment, we added an edge feedback module that does scatter an edge representation back to the pocket and the ligand, in a more restricted way than equivariant message passing does, namely it is symmetric, scalar-only, and without geometry (i.e. spherical harmonics), as described in the appendix. During exploratory experiments, we found that this mechanism does not improve the backbone itself. Because it wasn't included in the final set of modules + hyper parameters in the vanilla network, we did not report its five-fold CV or five-seed averages as an independent backbone model. It did, however, later improve the performance of paired readouts that operated on edges. We therefore used both persistent edges and edge feedback in PEAR and performed an ablation of it accordingly. The bound and unbound copies still produce distinct representations due to message passing at the interface in the bound copy. However, we recognize that this can on the surface be perceived as contrary to the physical heuristic that we allude to throughout the paper, i.e. a strictly unbound copy of the complex should have zero communication across the interface. However, the basis for using the unbound copy comes from its common usage as a data processing step in the literature and their empirical benefits, and we believe any improvements to the backbone that still maintains different representations between copies should be utilized, even if they risk infringing on the bound/unbound distinction on a heuristic level. Indeed, unlike physical symmetries, we have no mathematical need to enforce a specific definition of the unbound copy. For example, using real apo conformations is outside the scope of this paper (and more involved considering datasets such as apobind do not have full PDBBind coverage), but if we were to use them, we can still apply our edge feedback mechanism. In the final draft, we will make it clear that 1. the edge feedback mechanism does exchange some information across the interface in a more restricted way than equivariant message passing and 2. the ablation suggests the edge paired readout does not perform as well if there is no edge feedback.

Please continue to the next comment for the second part of our response.

3. On the preliminary experiment. The purpose of our preliminary experiment (Table 1) was to show that the vanilla MPNN does not benefit from the extra unbound copy in the CleanSplit dataset. We believe the experiment fulfills this purpose and provides a basis for our exposition on paired readouts. The backbones within this experiment are matched to each other in terms of architecture and number of parameters (only the head MLP input widths differ by necessity*) to isolate the difference between a one-copy network and a two-copy network over a small range of ℓmax⁡\ell_{\max} and layer values, so they are consistent internally. The difference between the vanilla neural networks and the backbone of PEAR is that PEAR has an additional edge feedback mechanism on persistent edge states during message passing. We include this in our ablation because "decontaminated" datasets such as CleanSplit are relatively new territory, so we believe identifying basic components of graph neural networks that improve performance is beneficial to the field. However, we acknowledge that this produces a logical fork in the sense that the benefits of paired readouts may appear to be overstated. We have since confirmed our exploratory finding by running the backbone with edge states and edge feedback but no paired readouts.

BackbonePaired readoutsParamsCASF-2016 RMSEpaired Δ\Delta vs PEAR95% CI
vanillano1,345,2331.323+0.065[+0.025, +0.109]
vanilla + edge states and feedbackno1,497,7451.371+0.113[+0.076, +0.151]
vanilla + edge states and feedback (PEAR)yes1,682,8971.258——

The first two rows share the absence of paired readouts, so their difference would isolate the edge mechanism, which costs 0.048 RMSE, 95% CI [+0.014, +0.081]. The second and third rows share a backbone, so their difference isolates the paired readouts, which are worth 0.113 RMSE, 95% CI [+0.076, +0.151]. These values are derived five fold CV ensembles, which exhibit a gain in accuracy from ensembling and so are in general lower than numbers in the ablation table.

PEAR outperforms the no-paired-readout variation over ℓmax⁡\ell_{\max} values 0, 1 and 2.

* the two-copy prediction head takes twice the number of input channels and hence has ~3.5-8% more parameters than the one-copy network depending on the number of layers.

Please continue to the next comment for the third part of our response.

4. On added capacity. It is true that a candidate reason for improved performance on these tasks is simply more capacity through added parameters in the paired readouts. Several networks in our submitted draft already do not perform better than models with fewer parameters. The two-copy networks in Table 1 have 3.5-8% more parameters than their one-copy counterparts. The "no-paired-readout" models in Table 3 gain 4.5% and 6.7% parameters as ℓmax⁡\ell_{\max} goes from 0 to 1 and 2, with RMSE of 1.386, 1.391 and 1.386.

  1. Because paired readouts add 25% more parameters, we obtain five fold CV RMSE and R values for a backbone model that has increased width to match the number of parameters of PEAR (1.68M). The model actually performed no better than the ensemble original backbone, at an RMSE of 1.347 vs. an RMSE of 1.323. Together with a PEAR model with intermolecular edges removed from the bound copy (RMSE of 1.389), these models are the three worst five-fold ensembles we've trained. Please see the following table:
ModelPaired readoutsParamsCASF-2016 RMSER
PEARyes1,682,8971.2580.834
Backbone, width-matched to PEARno1,682,7871.3470.803
PEAR, intermolecular edges removed from bound copyyes1,682,8971.3890.776
Backbone, original widthno1,345,2331.3230.814

All rows are five fold CV ensembles on CASF-2016.

5. On bound-only and MLP control. We have run a parameter-matched experiment where the atom paired readout is replaced by a per-node MLP on only the bound copy of the complex. It reaches a 5-fold CV RMSE of 1.276 against 1.258 for PEAR as reported. The Pearson R is 0.830 against 0.834 in PEAR. Both paired 95% CIs include zero; see below.

  1. As one can see, a per-node MLP on the bound copy can perform similarly well on cleansplit. For our interpretation of this result, see item 2.1.3 of our global comment, "On the fit of our work within the binding affinity prediction field".
ParamsCASF-2016 RMSEPearson R
PEAR1,682,8971.2580.834
Bound-only per-node MLP1,688,6181.2760.830
paired difference+0.3%+0.018−0.004
95% CI—[−0.005, +0.043][−0.014, +0.006]

Paired bootstrap over the 285 CASF-2016 complexes, 10,000 resamples.

6. On the fit of our work within the binding affinity prediction field. Please see our global comment.

Reviewer_xdRHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper motivates the exploration of using the unbound copy of the protein pocket as additional input to a model predicting protein ligand binding affinity. To compare equivariant features from the bound and unbound protein pocket representation are combined with Clebsch-Gordan products. The authors focus their evaluation on ensuring minimal test-train overlap by evaluating on three protein-ligand binding affinity splits (CleanSplit, ATOM3D LBA, CASF-2016 independent subset).

Critical strengths

Authors introduce paired outputs of atom, edge, and pocket level representations of bound and unbound states. They also introduce combining these representations with higher order Clebsch-Gordan products at the atom, and pocket level to improve representations used for binding affinity prediction.

Critical weaknesses

Performance improvements are in general limited in relative terms and lacking confidence intervals and seed variance in table 2, 4, and 5. On CleanSplit CASF-2016 RMSE the relative improvement is 1.26% and for the independent subset for RMSE it is worse by 4.97%. while the residue model is worse by 2.67%. In Table 4, pre-FT average person is worse by 7.43%, while improvement is 5.45% after FT-25. On ATOM3D LBA30 RMSE the improvement is 0.23%.

Limited demonstration that Clebsch-Gordan density products outperform other paired readouts in Table 3. For CleanSplit test RMSE in table 3 there is a 0.31% relative improvement over ℓmax=2\ell_\text{max} = 2 using just the difference Δ\Delta. Here the difference of Clebsch-Gordan products result in very similar performance compared to the difference of paired readouts. Additionally for models at the same ℓmax\ell_\text{max} the Δ\Delta approach outperforms the Clebsch-Gordan density products approach for ℓmax=1,2\ell_\text{max}=1,2.

Across Table 3,4,5 many configurations of PEAR are used at different ℓ\ell and atom and residue level. There does not seem to be a consensus of which mechanisms result in unified improvement across all datasets. For example, ℓ=2\ell=2 is necessary for highest performance on finetuning but for pre-finetuning the atom level approach does best. There does not appear to be a unified approach to use PEAR across these datasets.

Difficult to determine if improvement is because of the Clebsch-Gordan products operation or atom density. To address the authors should include a density only representation in Table 3 with other simple combinations.

Minor: Citations are missing in text for baselines used throughout the tables: GenScore, GEMS, BA-Pred, ATOMICA, Uni-Mol, ProFSA, LEFTNet, etc.

In Table 2 and throughout the paper, the independent subset should be clarified by referring to it as the CASF-2016 independent subset rather than as independent set so it is clearer what the source of this test set is.

Questions for authors

Can the authors please provide uncertainty estimates for Table 2, 4, and 5 in the main text. The improvements are limited in relative size.

The density representations appear to be quite useful in Table 3, could the authors please compare their Clebsch-Gordan products to simpler combinations of the density representations.

Is there a unified way to select the PEAR hyperparameters for binding affinity prediction? It appears in each of the benchmarks that differ hyperparameters are needed. Can the authors please report one fixed approach for all benchmarks and compare against baselines.

Discrepancy between pre-finetuning and finetuning, why does PEAR perform worse than GEMS+ATOMICA on pre-finetuning. Can the authors provide justification for why fine-tuning results in additional improvement to PEAR but not baselines?

3 comments

We thank the reviewer for the detailed comments.

1. On the reporting scheme for 5-fold CV values. We have set out our revised reporting scheme, including the withdrawal of the state-of-the-art wording on CleanSplit, in our global comment.

2. It is true that the mechanisms for pairing different representations can be interchanged to some extent. For example, in our paper on the dataset Cleansplit there are five varieties, specifically Δ, ‖Δ‖², ⟨hb,hub⟩, Δsym+MLP and Δsym, that perform within one standard deviation of each other at ℓmax = 1.

3. On a unified approach for all baselines and the lack of consensus between datasets.

  1. To illustrate why a lack of consensus between datasets is not an issue, consider the basis for developing new models that generalize better than models in the literature: there is already a lack of consensus between models trained on classical PDBBind and their performance on decontaminated versions of PDBBind.
  2. We agree that there is no single consensus across all datasets on a hyper parameter set. This can be expected from how the datasets were constructed. In brief, CleanSplit removes physically similar examples of the complex based on the Tanimoto similarity, ligand RMSD and TM-score from the training set. The seven held out clusters that we performed finetuning hold out entire protein families based on a 50% lDDT community threshold, i.e. from pocket structure similarity. LBA removes protein sequence similarity and is a much smaller dataset. In a broader context, the motivation for developing generalizable models is that existing models do not perform well on splits that remove information leakage. As such, we expect that models would also perform differently depending on how that information leakage is removed and to what extent. One pattern in our work is the spatial resolution of the graph: residue models seem to work better on test sets consisting of the more difficult held out clusters whereas atom models seem to work better on CleanSplit. This already presents a split of headline models into residue and atom variants. We believe it is valuable to provide such variants if they can meet or exceed other model performances in the literature, many of which have the advantage of synthetic pretraining or pretrained language models.
  3. For the ⊗ρCG model, the same hyper parameters were used for all tasks in the paper, which provides a fixed approach that is consistently better than the backbone.
  4. We applied the same residue ⊗ρCG model from the first two tasks to LBA30 and LBA60, and saw an inferior performance compared to the atom models. Similar to the PEAR-LBA, we can conceivably come up with a variation of this model PEAR-LBA-residue that improves upon these numbers by tuning its paired readout modules. Such improvements would be worth including in the paper even if they deviate from the hyper parameters used to perform the first two tasks.

4. On the independent subset. An interesting detail about cleansplit is that, in principle, independent subset is actually not a strictly harder subset than the CASF 2016 test set. As we understand it, CleanSplit's protocol to construct its from the PDBBind general set already removes training examples that are structurally similar analogues to those in CASF 2016. Such analogues are computed equally for all 285 examples in the test set. In the final version of the paper we will call it the CASF-2016 independent subset.

Please continue to the next comment for the second part of our response.

5. On PEAR being worse on the independent subset than GEMS. Bootstrapping the independent subset shows that the deficit of PEAR is within measurement error (see table below). However, we acknowledge that the mean RMSE and R being less favorable should be addressed. We offer one hypothesis: similar to how PEAR did not perform well relative to GEMS on the zero-shot task on held-out clusters, pretrained language models may provide an inherent advantage in this test set. To examine this we computed the nearest-neighbor distance of each test complex to the CleanSplit training set in four pretrained embedding spaces. The independent complexes are significantly farther from training in ESM2 protein-sequence space, while the two ligand-side spaces (ChemBERTa and ATOMICA) show no difference. A second sequence embedding, ANKH, has the same trend but not significantly. GEMS incorporates ESM and ANKH in their residue representations, so these complexes are plausibly closer to its training distribution than to ours.

The RMSE and R table in Kopko et al. comes from a single draw of 25 examples. For fair comparison we provided our values using the same format in the paper. However, we report that the average finetuning performance of the models vary by about ±0.02\pm 0.02 over five independent draws, and each PEAR model remains ahead of their backbone on average over these draws.

  1. We are not claiming that the current iteration of PEAR can benefit directly from ANKH and ESM2 since the purpose of designing PEAR was to identify inductive biases beneficial to generalization without pretrained representations.

Independent subset (144 complexes), 10,000 bootstrap resamples:

ModelRMSE95% CIPearson95% CI
PEAR1.414[1.260, 1.561]0.822[0.773, 0.863]
PEAR$_{\mathrm{residue}}$1.383[1.237, 1.530]0.833[0.786, 0.873]
joint ensemble1.368[1.225, 1.507]0.845[0.804, 0.881]
GEMS1.347———

GEMS lies inside all three RMSE intervals. Its CI cells are empty because per-complex predictions are not available in its deposit, so it can be neither bootstrapped nor paired against.

Nearest-neighbour distance to the CleanSplit training set:

EmbeddingModalityIndependent (144)Non-independent (141)p
ESM2 (1280d)protein sequence0.54 ± 0.460.36 ± 0.181.0e-5
ANKH (768d)protein sequence0.23 ± 0.100.21 ± 0.110.078
ChemBERTa (384d)ligand SMILES1.53 ± 1.111.33 ± 0.820.27
ATOMICA (32d)complex graph0.010 ± 0.0050.011 ± 0.0061.0

6. To answer your question on where the density expansion seems to help the network generalize, we found that CleanSplit performance doesn't perfectly predict performance on other datasets. Empirically the ⊗CGρ\otimes^{\rho}_{\mathrm{CG}} (ℓ=0\ell{=}0) network performs similarly to ⊗CG\otimes_{\mathrm{CG}} (ℓ=1\ell{=}1) on Cleansplit, 1.293 against 1.303, but outperforms it on the more difficult held out clusters, with mean pre-finetuning R of 0.452 against 0.419.

7. Resolution of the graph. There are many possible reasons that one resolution of the protein graph can perform better on one dataset and worse on another. A proposed mechanism is that higher resolution representations such one that represents all heavy atoms has limited utility when the test set is very dissimilar to the training set, and in such cases a lower resolution representations, such as using alpha-carbons as nodes, removes the network's commitment to the exact arrangement of the atoms. In contrast, in the classic PDBBind dataset which has information leakage between training and test sets, most models in the literature that utilize protein graphs usually resort to the all atom representation. GEMS, which was trained on CleanSplit, used the residue graph. We view this as an interesting result, since the residue model consistently performs better when protein families are held out entirely.

Please continue to the next comment for the third part of our response.

8. On choosing which operation to use in a paired readout. In the design process of our architecture we answered a very concrete question after noticing that the bound and unbound representations are usually not fully utilized, or are only fused after the learned representations have been fully constructed. That is, what are the operations available in an equivariant framework that takes two representations as input? There is a lot of freedom in choosing such operations, including the dot product, MLP and Clebsch-Gordan products which in a neural network can all fulfill the role of a nonlinearity at certain stages. Hence a priori, especially on a scalar backbone, we don't favor one type over another. That we ended up choosing the CG product based on empirical results does not preclude the possibility that there exist better parameterizations based on other operations, for example those that are fully scalar or higher order. Furthermore, it was correctly noticed in the review that the dot product is indeed the ℓ=0\ell=0 part of the Clebsch-Gordan product. However, in practice they are reduced to scalars differently. In our model the CG product's output is contracted against a third factor, namely the unbound representation, to keep (ℓ1,ℓ2)→ℓmax⁡(\ell_1,\ell_2) \to \ell_{\max} outputs of the CG product. The dot product is perhaps the more intuitive operation, but it does discard all higher-ℓ\ell outputs of the CG product. This is another example where we did not know which approach would work better before running experiments. From this perspective we view our contribution as exposing the possible operations that can be in a paired readout and finding parameterizations that generalize well beyond what we could achieve with a vanilla message passing network. In particular, every one of the 24 networks with paired readouts in Table 3 achieves a lower RMSE than networks without, with no overlap between their ranges of RMSE.

9. On density vs. product in ⊗CGρ\otimes^{\rho}_{\mathrm{CG}}. We agree that including both the density in CG product in one operation can obscure their contributions individually.

  1. We provide the performance of the CG product paired readout without densities, and the performance of non-CG product paired readouts.
Atom paired readoutℓ=0\ell{=}0ℓ=1\ell{=}1ℓ=2\ell{=}2
⊗CGρ\otimes^{\rho}_{\mathrm{CG}} (CG product on densities)1.293 ± 0.0151.344 ± 0.0211.316 ± 0.012
⊗CG\otimes_{\mathrm{CG}} (CG product, no densities)1.352 ± 0.0121.303 ± 0.0141.315 ± 0.011
Δ\Delta1.326 ± 0.0201.313 ± 0.0321.297 ± 0.012
Δsym\Delta_{\mathrm{sym}}1.342 ± 0.0371.326 ± 0.0181.338 ± 0.014
∥Δ∥2\lVert\Delta\rVert^21.360 ± 0.0341.318 ± 0.0201.329 ± 0.010
⟨hb,hub⟩\langle h_b, h_{ub}\rangle1.371 ± 0.0251.324 ± 0.0361.326 ± 0.012

CleanSplit test RMSE, mean ± standard deviation over five seeds, reproduced from Table 3. 2. The density readout is redundant on non-CG product operations because there would be no route to produce new higher-ℓ\ell terms. The effect of re-embedding the output representations into a density would be reversed when they are scalarized. 3. The density readout is also redundant without paired readouts because it is identical to the input data.

Reviewer_MyebOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how an equivariant message passing network trained from scratch can exploit the unbound state of a protein-ligand complex. It proposes PEAR, which adds paired readouts combining bound and unbound embeddings at atoms, intermolecular edges and the pocket. It is evaluated on PDBbind CleanSplit/CASF-2016, ATOM3D LBA, and seven held-out pocket clusters with few-shot finetuning.

Critical strengths

The paper asks a well-motivated question that complements the trend toward pretraining: which inductive biases help an equivariant MPNN trained from scratch on leakage-controlled splits. It supports its design with an honest negative result showing that naively pooling the unbound state does not help, together with a modular taxonomy of equivariant operations for combining bound and unbound embeddings. Empirically, PEAR achieves the best CASF-2016 accuracy on CleanSplit without pretraining and is competitive on ATOM3D LBA and the held-out pocket clusters. The multi-seed ablations and careful reporting, including the acknowledgment that the LBA30 gains are within one standard deviation, further strengthen the work.

Critical weaknesses

The headline gains are small relative to the noise, and the selection process is not clean. In Table 2, PEAR leads GEMS by 0.016 RMSE on CASF-2016, with no variance reported for the ensembles. In Table 3, many ablation differences lie within standard deviation 1. The chosen configuration was picked from a sweep of roughly 24 settings by the lowest mean RMSE on the same CleanSplit evaluation, and the l_max trends are non-monotonic (the same readout gives 1.293, 1.344 and 1.316). This weakens the claim that PEAR is the most accurate model and that "any" component removal reliably hurts. Significance testing or selection on a validation split would resolve this.

Questions for authors

Are the ensemble results in Table 2 and the Table 4 selections made on held-out data? How many configurations were tried, and is the chosen variant picked on the test clusters? Please report standard deviations or bootstrap intervals for the CASF-2016 ensembles.

1 comment

5. On the preliminary experiment. The purpose of our preliminary experiment (Table 1) was to show that the vanilla MPNN does not benefit from the extra unbound copy in the CleanSplit dataset. We believe the experiment fulfills this purpose and provides a basis for our exposition on paired readouts. The backbones within this experiment are matched to each other in terms of architecture and number of parameters (only the head MLP input widths differ by necessity*) to isolate the difference between a one-copy network and a two-copy network over a small range of ℓmax⁡\ell_{\max} and layer values, so they are consistent internally. The difference between the vanilla neural networks and the backbone of PEAR is that PEAR has an additional edge feedback mechanism on persistent edge states during message passing. We include this in our ablation because "decontaminated" datasets such as CleanSplit are relatively new territory, so we believe identifying basic components of graph neural networks that improve performance is beneficial to the field. However, we acknowledge that this produces a logical fork in the sense that the benefits of paired readouts may appear to be overstated. In Table 3 we provide a variation with "all paired readouts removed". Because the persistent edge states serve only the edge paired readout, this variation also loses edge feedback, so it has the vanilla backbone. We have since run the backbone with edge states and edge feedback but no paired readouts.

BackbonePaired readoutsParamsCASF-2016 RMSEpaired Δ\Delta vs PEAR95% CI
vanillanone1,345,2331.323+0.065[+0.025, +0.109]
vanilla + edge states and feedbacknone1,497,7451.371+0.113[+0.076, +0.151]
vanilla + edge states and feedbackall four (PEAR)1,682,8971.258——

The first two rows share the absence of paired readouts, so their difference isolates the edge mechanism, which costs 0.048 RMSE, 95% CI [+0.014, +0.081]. The second and third rows share a backbone, so their difference isolates the paired readouts, which are worth 0.113 RMSE, 95% CI [+0.076, +0.151]. These values are five fold CV numbers, which exhibit a gain in accuracy from ensembling and so are in general lower than numbers in the ablation table.

PEAR outperforms the no-paired-readout variation over ℓmax⁡\ell_{\max} values 0, 1 and 2.

* the two-copy prediction head takes twice the number of input channels and hence has ~3.5-8% more parameters than the one-copy network depending on the number of layers.

6. On added capacity. It is true that a candidate reason for improved performance on these tasks is simply more capacity through added parameters in the paired readouts. Several networks in our submitted draft already do not perform better than models with fewer parameters. The two-copy networks in Table 1 have 3.5-8% more parameters than their one-copy counterparts. The "no-paired-readout" models in Table 3 gain 4.5% and 6.7% parameters as ℓmax⁡\ell_{\max} goes from 0 to 1 and 2, with RMSE of 1.386, 1.391 and 1.386.

  1. Because paired readouts add 25% more parameters, we obtain five fold CV RMSE and R values for a backbone model that has increased width to match the number of parameters of PEAR (1.68M). The model actually performed no better than the ensemble original backbone, at an RMSE of 1.347 vs. an RMSE of 1.323. Together with a PEAR model with intermolecular edges removed from the bound copy (RMSE of 1.389), these models are the three worst five-fold ensembles we've trained. Please see the following table:
ModelPaired readoutsParamsCASF-2016 RMSER
PEAR (reported)all four1,682,8971.2580.834
Backbone, width-matched to PEARnone1,682,7871.3470.803
PEAR, intermolecular edges removed from bound copyinactive1,682,8971.3890.776
Backbone, original widthnone1,345,2331.3230.814

All rows are five fold CV ensembles on CASF-2016.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.