- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces two new ideas to assist with protein property prediction from tokenization. The first is RamaBPE, a byte pair encoding strategy that uses a discrete (k-NN) tokenization of Ramachandran angles, compressed using byte pair encoding to represent local structural motifs in proteins. The second is a hyperbolic representation of this feature, which, together with a custom loss function, enables a compact representation of nested properties such as protein substructures. The authors combine this with an ESM3 representation to achieve strong performance on a variety of protein structure tokenization classification tasks such as functino prediction, remote homology, and flexibility.
Critical strengths
The underlying ideas in this work are interesting, and I think there is promise in further developments in protein structure token compression. RamaBPE is a sensible and innovative step which is different from GeoBPE and is potentially extremely fast at inference time. The hyperbolic representation is an interesting and logical idea, which seems to improve performance. There were some ablations and analysis of the impact of the underlying geometry.
Critical weaknesses
I have serious concerns about data leakage in the evaluations. ESM3 was trained on protein function labels and its embeddings are being used to train classifiers for the same labels. This explains the apparently spectacular results on protein function and remote homology detection. The pure ESM3 results listed in the table, which appear to serve as an ablation of removing the MORPH contributions, use only the ESM3 structure encoder (not the full embeddings). Until this is addressed I would strongly advocate rejecting this paper.
I appreciated that the authors noted the Euclidean/hyperbolic ablation was not a pure ablation on the representation geometry since it also has different training losses. I would like to see more ablations, in particular a pure swap between Euclidean and hyperbolic, as well as ESM3-only and possibly swapping RamaBPE with GeoBPE. Currently, it is very hard to understand the impact of each technical contribution.
Questions for authors
What motivated the swap between hyperbolic and Euclidean loss functions for ordering parents closer/farther from the origin? It seems to me that the Euclidean one potentially makes more sense if smaller components piece together to build larger motifs, but I may be missing something about hyperbolic geometry.
Did you try any other tests related to Figure 3? I am very interested in both the impact of the encoding and the geometry, but I don't quite understand the metric used for (B). Is this meant to project everything onto the unit sphere? If so, surely you are losing ability to seperate the clusters in space?
Small point, but is the inference performance vs GeoBPE substantially different? It strikes me that RamaBPE would be extremely efficient once the k-NN and BPE are trained.