acceptodds
Under review as a conference paper at ICLR 2027

Hyperbolic Protein Representation Learning with Shared Structural Motifs

Abstract

Protein structures reuse local geometric patterns across diverse proteins, providing a natural basis for sharing structural knowledge. Exploiting this reuse requires representations that capture both multiscale composition and geometric variation among motif occurrences. We introduce MORPH (MOtif Relations for Protein representations in Hyperbolic space), a framework for learning hyperbolic protein representations through shared structural motifs. MORPH uses RamaBPE to construct a reusable vocabulary of variable-length motifs and their ordered multiscale compositions from backbone geometry. Hyperbolic entailment constraints organize compositional relations among both shared motif types and protein-specific occurrences, while explicit type-to-occurrence instantiation loss connects occurrences to their corresponding types. Complementary supervision derived from three-dimensional structures captures continuous geometric relations among occurrences. These signals train a mixture-of-curvature adapter over frozen ESM3 representations, combining Euclidean and Lorentz experts to produce hyperbolic residue representations for downstream prediction. MORPH demonstrates strong downstream transfer across functional site prediction, structural flexibility prediction, and protein-level fold classification. Analysis of motif occurrence representations further reveals composition-related radial organization in hyperbolic space. These findings support shared motif relations and occurrence-level geometric supervision as an effective basis for adapting pretrained protein representations.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
protein language models, hyperbolic representation learning, mixed geometry, hierarchical representation learning, structure-guided learning, remote homology retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nwUnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces two new ideas to assist with protein property prediction from tokenization. The first is RamaBPE, a byte pair encoding strategy that uses a discrete (k-NN) tokenization of Ramachandran angles, compressed using byte pair encoding to represent local structural motifs in proteins. The second is a hyperbolic representation of this feature, which, together with a custom loss function, enables a compact representation of nested properties such as protein substructures. The authors combine this with an ESM3 representation to achieve strong performance on a variety of protein structure tokenization classification tasks such as functino prediction, remote homology, and flexibility.

Critical strengths

The underlying ideas in this work are interesting, and I think there is promise in further developments in protein structure token compression. RamaBPE is a sensible and innovative step which is different from GeoBPE and is potentially extremely fast at inference time. The hyperbolic representation is an interesting and logical idea, which seems to improve performance. There were some ablations and analysis of the impact of the underlying geometry.

Critical weaknesses

I have serious concerns about data leakage in the evaluations. ESM3 was trained on protein function labels and its embeddings are being used to train classifiers for the same labels. This explains the apparently spectacular results on protein function and remote homology detection. The pure ESM3 results listed in the table, which appear to serve as an ablation of removing the MORPH contributions, use only the ESM3 structure encoder (not the full embeddings). Until this is addressed I would strongly advocate rejecting this paper.

I appreciated that the authors noted the Euclidean/hyperbolic ablation was not a pure ablation on the representation geometry since it also has different training losses. I would like to see more ablations, in particular a pure swap between Euclidean and hyperbolic, as well as ESM3-only and possibly swapping RamaBPE with GeoBPE. Currently, it is very hard to understand the impact of each technical contribution.

Questions for authors

What motivated the swap between hyperbolic and Euclidean loss functions for ordering parents closer/farther from the origin? It seems to me that the Euclidean one potentially makes more sense if smaller components piece together to build larger motifs, but I may be missing something about hyperbolic geometry.

Did you try any other tests related to Figure 3? I am very interested in both the impact of the encoding and the geometry, but I don't quite understand the metric used for (B). Is this meant to project everything onto the unit sphere? If so, surely you are losing ability to seperate the clusters in space?

Small point, but is the inference performance vs GeoBPE substantially different? It strikes me that RamaBPE would be extremely efficient once the k-NN and BPE are trained.

Reviewer_ojKjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MORPH, a framework for adapting pretrained protein representations through shared structural motifs. RamaBPE discretizes backbone torsions into Ramachandran states and applies ordered BPE merges to construct variable-length motifs and encode their compositional relations. The method distinguishes shared motif types from protein-specific occurrences and uses hyperbolic entailment constraints to model composition at the type level, composition at the occurrence level, and type-to-occurrence instantiation. Structure-derived similarity supervision further captures continuous geometric variation among occurrences. These objectives are used to train a mixture-of-curvature adapter with Euclidean and Lorentz experts on top of frozen ESM3 features, yielding residue representations for downstream prediction. Evaluation spans 12 tasks and 24 test splits covering functional-site prediction, structural flexibility prediction, and protein-level fold classification. The paper reports the highest scores among the compared methods on 15 splits and improvements over GeoBPE-Transfer on 18 splits. Additional analyses examine motif-construction cost, within-type geometric consistency, and span-length-dependent radial organization, together with a supplementary comparison against a Euclidean adapter trained with different objectives.

Critical strengths

1.A coherent formulation of shared motif identity and occurrence-specific variation.The distinction between motif types and concrete occurrences addresses a meaningful modeling issue: recurring structural patterns should support knowledge sharing without requiring their individual occurrences to have identical representations. MORPH combines type-level and occurrence-level compositional relations with explicit type-to-occurrence instantiation, while continuous geometric supervision preserves differences among occurrences. In particular, using cone containment rather than strict equality provides a coherent way to connect shared identity with instance-level variation. This formulation is arguably the most compelling methodological contribution of the paper. 2.Promising empirical results across several prediction settings.The reported improvements are not confined to a single task: MORPH exceeds GeoBPE-Transfer on 18 of 24 splits, including all six flexibility-prediction splits and all three protein-level classification splits. Some gains are substantial, such as the Homo Superfamily macro-F1 increase from 0.4728 to 0.9139 and the Ept Fold AUROC increase from 0.6478 to 0.7973. Although the comparison limitations discussed below require clarification, the breadth and magnitude of the reported improvements provide encouraging evidence for the overall framework.

Critical weaknesses
  1. The incremental benefit over a matched frozen ESM3 baseline is not adequately established. Appendix C.1 describes the sequence-and-structure inputs to ESM3-SM-open-v1 and the cached 1536-dimensional post-LayerNorm features. Although Table 1 includes an ESM3 column, its reference scores are taken from prior work, and the paper does not establish whether they use the same input modalities, feature extraction, and downstream evaluation pipeline as MORPH. A direct evaluation of the unadapted cached ESM3 features under an identical downstream pipeline would therefore be important. The downstream predictor architectures, whether the adapter is frozen during downstream training, the protein-level aggregation for Homo, and the downstream model-selection procedures are also insufficiently specified. Because a central empirical claim concerns the effectiveness of adapting pretrained protein representations, these omissions make it difficult to determine how much of the reported improvement is attributable to MORPH rather than to differences in feature extraction or evaluation.
  2. The experiments do not disentangle the contributions of relational supervision and hyperbolic geometry. The Euclidean comparison changes both representation geometry and training objectives. As detailed in Appendix A.1, it does not include the same explicit occurrence-hierarchy and type-to-occurrence constraints, uses a different radial convention, and introduces additional composition-prediction and pretrained-output-preservation losses. Although Appendix A and the conclusion acknowledge that this comparison does not isolate the effect of geometry, Section 4.4 nevertheless attributes the observed gains to properties inherent to hyperbolic space, including entailment cones. The advantage over the Euclidean configuration on 21 of 24 splits therefore does not establish which of these modeling choices is responsible. Moreover, the paper does not report controlled ablations that remove LgeoL _{geo} , LontL _{ont} or LinstL _{inst} separately. Figure 3 illustrates span-length-related separation but does not directly verify the intended parent–child entailment relations. Consequently, the individual contributions of the proposed supervision mechanisms and geometric formulation remain insufficiently demonstrated, limiting the evidence for the paper's central methodological claims.
Questions for authors
  1. Matched frozen-backbone baseline . What is the downstream performance of the unadapted ESM3-SM-open-v1 features when using exactly the same sequence-and-structure inputs, cached post-LayerNorm representations, and downstream training pipeline as MORPH? Please also clarify whether the ESM3 and GeoBPE-Transfer reference scores in Table 1 use matching backbone features and input modalities. Such a comparison would help establish the incremental benefit of the proposed adaptation beyond the pretrained representations.
  2. Downstream evaluation protocol.Could the authors specify the downstream predictor architectures, whether the MORPH adapter is frozen or fine-tuned during downstream training, how the Lorentz representations are supplied to the predictors, and how residue representations are aggregated for Homo? Please also describe the hyperparameter and checkpoint selection procedures for the downstream predictors, including whether validation data are used, and clarify whether the downstream training sets, label budgets, and selection procedures are comparable to those of the reference methods.
  3. Contributions of the supervision terms. Starting from the full MORPH model, could the authors provide controlled ablations that separately remove (i) the occurrence-level geometry loss LgeoL _{geo} (ii) the complete ontology objective LontL _{ont} , and (iii) only the type-to-occurrence instantiation term LinstL _{inst} , while retaining the other ontology terms? These comparisons should keep the backbone features, remaining architecture, training data, and downstream evaluation protocol fixed. Results on representative tasks would help quantify the incremental contribution of each supervision component and assess their claimed complementarity.
  4. Interpretation of the Euclidean comparison. Appendix A and the conclusion acknowledge that the Euclidean configuration differs from MORPH in both representation geometry and training objectives. Given the differences in relational constraints, radial supervision, and auxiliary losses, what evidence supports attributing the observed gains specifically to hyperbolic geometry or entailment cones? A comparison preserving the same relation categories and comparable auxiliary supervision would help disentangle these factors. In the absence of such evidence, could the authors align the interpretation in Section 4.4 with the stated limitation and restrict the conclusion to the relative performance of the two evaluated configurations?
Reviewer_xPXROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MORPH, which learns hyperbolic protein representations on top of frozen ESM3 by exploiting reusable local backbone motifs. Overall, I find the idea interesting, the motivation clear, and the implementation thoughtfully developed. However, my main concerns are the lack of controlled attribution experiments and the uncertainty around baseline comparability.The current results support the complete MORPH configuration more strongly than they support the claim that hyperbolic motif relations are an effective basis for adapting protein representations.

Critical strengths

The problem is real, not invented. Local backbone geometry does recur across unrelated proteins, and the same motif appears at multiple scales with varying conformations. This is an objective property of 3D structure, and most successful structure-based methods already model it implicitly — AlphaFold's triangle/IPA machinery and its torsion-angle outputs are effectively exploiting the same regularity. Making the reuse and the composition relations explicit is a legitimate and interesting direction.

Critical weaknesses
  • The experiments do not sufficiently isolate the source of the gains. MORPH introduces several changes together: motif embeddings, structural similarity supervision, ontology losses, and a mixed-curvature adapter. Without more controlled ablations, it is difficult to tell which components are responsible for the improvements. In particular, the Euclidean comparison changes both geometry and training objectives, so it does not establish the benefit of hyperbolic geometry itself.

  • The downstream evaluation could be more targeted to the proposed geometric benefits. For example, FLIP–Meltome could test thermostability prediction, the EC/GO benchmarks used by GearNet could test protein-level functional transfer, and ATOM3D–LBA could test binding affinity rather than binding-site identification. A side-chain prediction experiment using the CASP15/16 setup adopted by DiffPack would provide a more geometry-focused diagnostic, using torsion-angle errors and side-chain RMSD. These are suggested directions rather than a request to run every benchmark.

Questions for authors

What is the training recipe? data split, filter etc.?

Could you please adding loss-level ablation, so nothing is attributable to the hyperbolic part?

Can you evaluate on dihedral-sensitive tasks (side-chain/rotamer or torsion prediction) plus stability and protein-level function? Relatedly, what explains Homo-SupFam jumping from 0.4728 to 0.9139 while Homo-Fold stays at 0.3355 — can leakage and probe-capacity differences be ruled out?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.