- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes EASER. It uses a frozen pretrained 3D GNN as the encoder, and, instead of using a standard neural readout, it uses am explicit symbolic pipeline for conformer ensemble property prediction of molecules. It uses cross fitted residualization to filter cheap descriptor signals, and go on to use genetic programming to explore further nonlinear features over the latent space. The paper separates 2 roles of conformer energy, one is the deterministic routing to energy-bin-specific symbolic experts, while the other is the learned Boltzmann-form aggregation of conformer scores. The method is evaluated on 6 tasks from MoleculeNet, against the vanilla neural net and other baseline models.
Critical strengths
-
The experiments are thorough and meticulous, covering major baselines and diverse tasks on MoleculeNet, with ablation studies and clean budget control.
-
The methods is physically well motivated, providing an interpretable readout that can be potentially useful for chemists to audit and justify a prediction.
Critical weaknesses
-
None of the discovered symbolic expressions are reported in the manuscript, and only aggregate structural statistics are provided. It is unclear what the actual scientific benefit would be of those symbolic expressions to a chemist.
-
There are no case studies or benchmarking on harder realistic molecular tasks or OOD settings, and everything is carried out on MoleculeNet. Also, "audit" would mean that by looking at the symbolic output, we can identify failure modes of a model, but this is also absent from the current version of the paper.
-
The recurrence metric on ClinTox (0.52) and BBBP (0.57) seems to suggest that the readout learned function varies a lot across the different runs, thus making a stable audit not plausible.
Questions for authors
-
There is a mismatch between two places reporting random-split FreeSolv: 0.89 in Table 1, and 0.96 in Sec 4.2. Which one is correct?
-
Can you provide a few examples of the symbolic expressions discovered by the protocol, and try to do a audit by interpreting those symbolic expressions?
-
Given the functional stability is only 0.52 on ClinTox, do you suggest a chemist to use this for reliably auditing the predictions? Can you also provide the prediction-level agreement across the GP seeds alongside the feature-level recurrence?
Thank you for your careful response, which has addressed some of my concerns. I am still not fully convinced that the reconstructed expressions would provide further insight into the underlying chemical principle guiding the system, or that they could shed further insights that can be verified by the chemists. I encourage the authors to involve chemists in the evaluation of the qualities of those expressions, and to provide a practical case study for how those expressions would enlighten chemists in a principled way.