acceptodds
Under review as a conference paper at ICLR 2027

Hyper-Fold: Exploring the Expressive Limit of Sequence-Geometry Learning for Proteins via Hypergraph Modeling

Abstract

Protein structure modeling rests on a single computational primitive: the interaction between what a residue *is* (sequence content) and *where it sits* (three‑dimensional geometry). What is the expressive limit of this layer class? We show that the complete bilinear operator over content‑geometry outer products—the sufficient statistic of all second‑order interactions—is the expressive ceiling, while the additive message passing of mainstream geometric GNNs is provably blind to content‑geometry binding. We then introduce **Hyper‑Fold**, a rank‑ separable convolutional backbone approaching this ceiling at message‑passing cost: each radius neighborhood is organized into a sequence hyperedge and a contact hyperedge, modulated by an edge‑conditioned matrix‑valued operator factorized into learned basis operators with geometry‑generated coefficients. Across enzyme function prediction, fold classification, and ligand binding site detection, **Hyper‑Fold** and its hierarchical variant **Hyper‑Fold‑Deep** achieve the best results among protein‑specific structure encoders; **Hyper‑Fold‑Pocket**, an anchored set‑prediction head, surpasses UniSite‑3D on UniSite‑DS and two zero‑shot benchmarks with no sequence language model features, fewer parameters, and lower latency—suggesting that a sufficiently expressive 3D backbone recovers information that fusion architectures previously borrowed from evolution‑scale pretraining.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Learning on Geometry, Hypergraph Neural Networks, Protein Learning, Object Detection, Pocket Detection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MQFqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an expressivity analysis of content–geometry interactions, showing that separable additive aggregation cannot distinguish changes in their pairing and characterising bilinear interactions through outer products. Motivated by this analysis, the authors introduce Fold-Conv, which uses GK-Net to combine learned basis matrices according to relative geometry and sequence separation. They build Hyper-Fold backbones and a dedicated pocket-detection head, reporting strong results on enzyme-function prediction, fold classification, and pocket detection with better computational efficiency.

Critical strengths
  1. An efficient convolution motivated by an expressivity analysis. The authors analyse an expressivity ladder from additive message passing to scalar-, channel-, and matrix-gated interactions, highlighting how these operators couple residue content with relative geometry. This analysis motivates Fold-Conv, which allows geometry to control both the weighting and mixing of residue feature channels. GK-Net generates coefficients conditioned on relative geometry and sequence separation to combine a small set of learned basis matrices, providing a practical implementation of matrix gating without generating a full transformation matrix independently for every edge.

  2. Strong empirical results across multiple tasks. Hyper-Fold achieves strong results on enzyme-function prediction, fold classification, and pocket detection. Hyper-Fold-Pocket also outperforms the evaluated baselines on two external pocket benchmarks, with fewer parameters and lower reported inference latency than UniSite-3D, without pretrained protein-language-model embeddings.

Critical weaknesses
  1. The theoretical expressivity analysis does not fully support the claimed expressivity hierarchy or the motivation for matrix gating. The authors use the expressivity analysis to motivate Fold-Conv, but the expressive-ceiling result applies to a restricted bilinear function class, and its extension to general multilayer geometric GNNs is not established. Moreover, distinguishing binding swaps does not by itself justify matrix gating, since simpler scalar and channel gates can already distinguish such configurations. Matrix gating allows richer geometry-dependent channel mixing, but the analysis does not clearly establish which additional interactions are important for protein modelling. This leaves a gap between the theoretical motivation and the specific operator proposed.

  2. The experiments do not sufficiently isolate the additional benefit of matrix gating. The operator comparisons use a shared architecture and training recipe, which is a useful control. However, channel gating accounts for much of the improvement, while matrix gating introduces additional parameters and the reported differences lack uncertainty estimates. It therefore remains unclear whether the incremental gains reliably reflect the richer interaction mechanism rather than increased capacity or variation across runs. Furthermore, the comparison with UniSite-3D changes both the backbone and pocket-detection head, making it difficult to attribute the overall improvement specifically to Fold-Conv. The authors need more controlled comparisons to establish the contributions of the operator and the detector design separately.

Questions for authors
  1. “Hypergraph modeling” in the title may be misleading. The authors explicitly state that hyperedges only organize sequence and spatial-contact neighborhoods, while Fold-Conv aggregates pairwise messages without hypergraph propagation.

  2. Scope of the expressivity analysis. Could the authors clarify the scope of Theorem 1 and its relationship to existing edge-conditioned message-passing formulations? In particular, how does the claim in Appendix F.3 that stacking preserves the bilinear class account for composition and nonlinear transformations?

  3. Controlled comparisons. In Table 3, matrix gating improves EC Fmax from 0.777 to 0.789 and pocket [email protected] from 0.432 to 0.467 over channel gating, while the reported classification-model parameter count increases from 3.6M to 6.6M. Could the authors provide parameter-matched comparisons with repeated runs and uncertainty estimates to establish whether these gains reflect the interaction mechanism rather than increased capacity? For pocket detection, could alternative backbones be evaluated under the same detection head to separate Fold-Conv’s contribution from the detector changes?

  4. Robustness to rotations. Could the authors clarify the orientation term in Equation 6 and verify rotational invariance of both the backbone and complete pocket model? With O(k)O^{(k)} defined as a frame-axis vector, the stated sum of element-wise products is not generally rotation-invariant. The pocket head’s Cartesian Fourier positional encoding also does not itself guarantee rotational invariance.

  5. Additional capability of matrix gating. Since scalar and channel gating can already distinguish binding swaps, could the authors clarify the contrary statement about scalar gating in Section 4.3? What additional interactions does matrix gating capture, and could the authors illustrate their importance with a concrete protein-modelling example?

7 comments
AuthorsOpenReview ↗

Response to Q1: Hypergraph Modeling

Hyper-Fold adopts a hypergraph modeling perspective to unify sequence relations and spatial-contact relations in proteins. Sequence neighborhoods and local spatial contacts are represented by hyperedges, and Fold-Conv models content–geometry interactions within these hyperedge-organized neighborhoods.

Thus, “Hypergraph Modeling” refers to the unified organization and modeling of multiple types of residue relations, rather than to one particular standard hypergraph convolution. Pairwise message computation is our specific implementation on this hypergraph structure. We have added this clarification to the Method and Related Work to make the connection between hyperedge-based relation organization and Fold-Conv message computation explicit.

AuthorsOpenReview ↗

Response to Q2: Scope of Theorem 1

Theorem 1 characterizes single-layer bilinear messages with a fixed geometry feature map: any bilinear map in $(x_j,\phi(\delta_{ij})) $ can be written as a linear readout of their outer product, and vice versa. This provides a common basis for comparing additive, scalar, channel, and matrix-gated interactions and motivates Fold-Conv's shared rank-KK basis implementation.

For multilayer networks, composition and nonlinearities generate new functional forms. For example, two successive interactions can produce xδ2x\delta^2, which lies outside the original bilinear class in (x,δ)(x,\delta). We therefore explicitly restrict the theorem to a single-layer characterization rather than a general expressivity ceiling for multilayer networks. We have added this scope clarification to the appendix and revised the abstract, Introduction, Figure 1, contributions, and Conclusion consistently.

We have also added direct comparisons with CDConv and other edge-conditioned message-passing methods in the Fold-Conv section and Related Work, highlighting our shared-basis parameterization, geometry-dependent cross-channel mixing, and unified sequence/contact-hyperedge organization.

AuthorsOpenReview ↗

Response to Q3: Controlled Comparisons and Backbone/Head Controls

We have added three independent runs to Table 3 under a shared backbone, head, neighborhood, and training recipe:

MetricChannelMatrix
EC Fmax@500.771 ± 0.0030.788 ± 0.001
EC AUPR0.868 ± 0.0050.871 ± 0.002
Pocket [email protected]0.604 ± 0.0020.615 ± 0.005
Pocket [email protected]0.440 ± 0.0020.470 ± 0.004

Figure 10 further expands the base width, increasing Channel to 5.5M parameters for EC and 8.9M for Pocket, comparable in scale to Matrix at 6.6M and 10.1M. At these similar scales, Matrix achieves EC Fmax@50/AUPR of 0.777/0.873, versus 0.771/0.869 for Channel, and Pocket [email protected]/[email protected] of 0.618/0.473, versus 0.606/0.447. Thus, the Matrix advantage persists after capacity expansion.

The new Table 4 separately evaluates backbone and detection-head contributions:

BackboneDETR-style head [email protected]/[email protected]Hyper-Fold-Pocket head [email protected]/[email protected]
GearNet0.420/0.2060.434/0.224
Hyper-Fold0.603/0.4540.610/0.473

Hyper-Fold outperforms GearNet under both detection heads, and our head improves performance with both backbones. These controls show that the backbone and detector each contribute to the final improvement.

Reviewer_QZKqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Hyper-Fold combines residue features with sequence-conditioned geometric features through outer-product interactions and a shared linear readout. It uses hierarchical pooling for protein-level prediction and preserves full-resolution features for pocket detection. The paper reports strong results across protein classification and pocket detection without pretrained protein language model features, supported by architectural ablations and theoretical analyses of expressivity and approximation error.

Critical strengths
  • Learning directly from amino-acid identities and structures without pretrained protein language model features addresses a useful setting. The reported performance across multiple tasks supports the approach’s potential as a practical protein representation model.

  • Preserving residue-level resolution for pocket detection while using hierarchical pooling for protein-level prediction is a sensible design choice. The architectural ablations provide supporting evidence for these choices and strengthen the empirical case for the framework.

Critical weaknesses
  • The paper contrasts Hyper-Fold’s factorization with CDConv’s memory-intensive matrix kernels. However, CDConv appears to already generate a low-dimensional geometric kernel vector, form an outer product with neighboring features, aggregate the resulting messages, and apply a shared linear readout. The precise algebraic and computational distinction therefore remains unclear. Moreover, the matrix-gating comparison in Table 3 uses 6.6M versus 3.6M parameters, approximately 83% more, so the performance difference cannot be cleanly attributed to interaction structure. These issues are central to assessing both the methodological novelty and the source of the reported gains.

  • For Theorem 1, the single-layer tensor-product argument appears sound, but it does not by itself establish the same expressive ceiling for stacked networks. With a fixed geometry map phi(delta) = delta, two layers can compute h = x * delta followed by y = h * delta = x * delta^2, which lies outside the original bilinear class in (x, delta). Any network-level ceiling therefore requires additional assumptions. The connection between optimal projection error and the error achieved by the trained model also needs clarification. Finally, if W_ij in Eq. (9) already incorporates W_v, the output difference is sum_j [(W_ij - W_hat_ij) * x_j]; the additional appearance of W_v in the proof of Corollary 2 then requires justification. These points affect whether the theory supports the paper’s central interpretation of the architecture.

Questions for authors
  • What precise algebraic or computational contribution distinguishes Hyper-Fold from CDConv’s apparent outer-product formulation and shared readout? Which differences explain the reported efficiency gains?

  • Does Theorem 1 apply only to individual layers with a fixed geometry feature map, or also to stacked networks? What assumptions support the latter interpretation, and how does optimal projection error relate to deployed-model error? In Corollary 2, does W include W_v, and should the feature-norm term consequently use the squared norms of the unprojected inputs x_j?

  • Can the authors provide comparisons that isolate the proposed interaction mechanism under matched backbone, head, neighborhood, training configuration, and parameter budget, with repeated runs to assess variability? In particular, does the Table 3 advantage persist after matching capacity?

  • Figure 4A: Is Hyper-Fold’s displayed prediction selected by the best ground-truth IoU among its top five predictions, while UniSite-3D’s is selected by the highest model score? If so, please regenerate the panel using identical selection rules. This concern applies specifically to the qualitative comparison.

6 comments
AuthorsOpenReview ↗

Response to Q1: Distinction from CDConv

Both Hyper-Fold and CDConv use outer products and a shared linear readout; the key distinction lies in the subsequent parameterization and message-passing structure. The computational paths can be written as:

\mathbf h_i^{\mathrm{HF}}=\sum_jP_c(\tilde{\mathbf g}_{ij}\otimes\mathbf W_v\mathbf x_j).$$ CDConv uses separate displacement-indexed parameters throughout its WeightNet. Fold-Conv instead uses displacement buckets only in the first GK-Net projection, followed by a shared MLP and basis operators, avoiding duplication of the full kernel network for every sequence offset. Fold-Conv also explicitly propagates over sequence and contact hyperedges. CDConv's published backbone pools residues hierarchically. When we removed pooling to obtain full-length residue features, the model ran out of memory under our training configuration. Hyper-Fold instead preserves full sequence length and directly supports residue-level pocket detection. | Method | Params | Latency | EC Fmax@50 | EC AUPR | | ---------- | -------------: | ----------------: | --------------: | --------------: | | CDConv | 31.0M | 14.2 ms | 0.777 | 0.859 | | Hyper-Fold | **7.7M** | **10.5 ms** | **0.789** | **0.874** | Hyper-Fold improves both EC metrics with approximately one-quarter of CDConv's parameters and lower latency. These whole-model measurements support the efficiency advantage of the complete architecture. We have added this comparison to the Fold-Conv section and Related Work.
AuthorsOpenReview ↗

Response to Q2: Theorem Scope, Projection Error, and Corollary 2

The reviewer's stacking counterexample is correct: with ϕ(δ)=δ\phi(\delta)=\delta, two successive layers can produce xδ2x\delta^2, which lies outside the original bilinear class in (x,δ)(x,\delta). We have removed the claim that stacking preserves this class by linearity and retitled Theorem 1 as “Single-layer characterization of bilinear interactions.” The theorem is now explicitly restricted to single-layer bilinear messages with a fixed geometry map, and serves as a characterization and motivation for the operator design rather than a general expressivity ceiling for arbitrarily deep networks. The abstract, introduction, contributions, Figure 1, conclusion, and appendix have been revised consistently.

We also distinguish optimal projection error from deployed-model error. The spectral tail characterizes the best rank-KK operator approximation; it does not bound the learned weight network's realized error or the trained model's optimization and generalization errors. We have clarified this distinction and retitled the relevant corollary “Relative operator error.”

Finally, because Wij\mathbf W_{ij} in Eq. (9) already absorbs Wv\mathbf W_v through the basis operators, the output difference is ∑jΔWijxj\sum_j\Delta\mathbf W_{ij}\mathbf x_j. We have corrected the propagation bound to use ∥xj∥22\|\mathbf x_j\|_2^2, rather than ∥Wvxj∥22\|\mathbf W_v\mathbf x_j\|_2^2, and unified the definitions of W\mathbf W, Bc\mathbf B_c, and Wv\mathbf W_v.

AuthorsOpenReview ↗

Response to Q3: Controlled Comparisons and Capacity Scaling

We have added three independent runs to Table 3, keeping the backbone, head, neighborhood, and training recipe fixed within each task. Matrix gating outperforms Channel gating on EC Fmax@50 (0.788±0.0010.788\pm0.001 vs. 0.771±0.0030.771\pm0.003), EC AUPR (0.871±0.0020.871\pm0.002 vs. 0.868±0.0050.868\pm0.005), Pocket [email protected] (0.615±0.0050.615\pm0.005 vs. 0.604±0.0020.604\pm0.002), and Pocket [email protected] (0.470±0.0040.470\pm0.004 vs. 0.440±0.0020.440\pm0.002).

We further varied the base width from C0 to C3, expanding the parameter budgets to approximately twice the native size. In particular, Channel gating was increased from 3.6M to 5.5M parameters for EC and from 7.0M to 8.9M for Pocket, bringing it to a comparable scale to Matrix gating at 6.6M and 10.1M, respectively.

The performance gap remains at these similar parameter scales:

TaskMetricChannelMatrix
ECFmax@500.7710.777
ECAUPR0.8690.873
Pocket[email protected]0.6060.618
Pocket[email protected]0.4470.473

Thus, the Matrix advantage persists when the two operators are compared at similar parameter scales. These comparisons have been added to Appendix Figure 10.

Reviewer_4WkbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

A key question in protein modeling is how to predict protein's properties from its 3D structure (obtained either from an experiment or from a predictive model). The paper provides a new GNN architecture for protein structures that leverages both sequence-level and 3D neighborhoods and uses some nice ways to define features on edges in the GNN neighborhood graph. The most important contribution is that the empirical results on a couple of protein benchmarks is very strong.

Critical strengths

The results are really good, and there is rigorous comparison to a number of recent baseline approaches. The exposition on GNNs and expressivity is accessible and useful.

Critical weaknesses

My main critique is that the 'theoretical' result seems fairly tautological. See below for details. Perhaps I'm misunderstanding, though.

Questions for authors

The results are very impressive. Your method provides strong performance improvements vs. recently published alternative methods. Can you help clarify which comparisons isolate just the impact of your message passing architecture. I'm assuming some used different featurizations for nodes/edges too? In those cases, it's hard to attribute the difference in performance to a particular source. I know you provide such ablations in Table 3, but it would be helpful to understand the structure of comparisons in the other table too.

I'm fundamentally confused by Section 2. Is Theorem 2 at all non-trivial? Isn't the theorem simply stating that any quadratic form can be expressed as a linear function of an outer product? This is just a fundamental property of outer products. There are claims across the paper about general expressivity ceilings. Is the claim that a quadratic function of an expressive feature representation (based on some kernel approximation, etc) is universal? As it stands, Section 2 feels like a way to get some extra math into the paper, but the result doesn't actually drive the rest of the paper. Please clarify.

I see a lot of argument about expressivity, but this is only half the story in machine learning. There is a fundamental bias-variance tradeoff. As you vary the expressivity, you also vary the ability learn the function on limited data reliably. Can you discuss the tradeoff? Why do you think your models performed best, despite being subject to limited training sets?

I have worked on EC classification before. I am impressed by the performance level. How does this compare to non-structure-based approaches on the same train-test split? I know there is a ton of literature on this.

The framing of proteins as graphs is not faithful to biological dogma. You say things like 'protein function is determined by three-dimensional structure' and ' a protein of L residues is a graph G'. These are high-level simplifications of proteins of mathematical convenience. A protein is a dynamic object subject to the laws of physics, etc. Please fix these.

6 comments
AuthorsOpenReview ↗

Response to Q1: Experimental Attribution

To address the reviewer's concern about attribution in end-to-end comparisons, it is important to recognize that the main-table comparisons often vary the backbone, detection head, node/edge featurization, or training strategy simultaneously; not all performance differences can therefore be attributed to Fold-Conv. We distinguish three levels of comparison in the revision: the main table evaluates the end-to-end competitiveness of complete systems; Table 3 changes only the aggregation operator under a shared backbone, head, neighborhoods, and training protocol to isolate the message-passing mechanism; and the new Table 4 further separates backbone and detection-head contributions.

Across three independent runs in Table 3, Matrix improves over Channel by 0.017 EC Fmax@50, 0.003 EC AUPR, 0.011 Pocket [email protected], and 0.030 [email protected], with sample standard deviations reported for each result. Table 4 provides the following controlled comparison:

BackboneDETR-style control [email protected]/[email protected]Hyper-Fold-Pocket head [email protected]/[email protected]
GearNet0.420/0.2060.434/0.224
Hyper-Fold0.603/0.4540.610/0.473

With the Hyper-Fold-Pocket head fixed, Hyper-Fold achieves [email protected]/[email protected] of 0.610/0.473 versus GearNet's 0.434/0.224; with the DETR-style control fixed, the corresponding values are 0.603/0.454 versus 0.420/0.206. Replacing the head with ours under a fixed backbone also improves Hyper-Fold by 0.007/0.019 and GearNet by 0.014/0.018. Thus, both the backbone and the head contribute independently; the end-to-end improvement does not arise from only one component.

In addition, Figure 5 and Tables 7–8 compare official UniSite-3D predictions with Hyper-Fold-Pocket across all 2,293 test proteins. The paired [email protected]/[email protected] differences are 0.0567/0.0835, with 95% whole-protein bootstrap CIs of [0.0450, 0.0678] and [0.0688, 0.0976], respectively; both improvements remain significant after Holm correction.

AuthorsOpenReview ↗

Response to Q2: Scope of the Theoretical Result

We agree with the reviewer that “any bilinear map can be represented as a linear function of an outer product” is a standard tensor-algebra property and should not itself be presented as a new mathematical theorem. Nor do we claim that arbitrarily deep networks are governed by the same bilinear ceiling, or that arbitrary quadratic feature representations automatically provide universal approximation. The original text overgeneralized the single-layer result into a network-level ceiling; we have narrowed the corresponding statements in the theorem, abstract, Introduction, Figure 1, and Conclusion.

The role of this standard algebraic property is to provide a common coordinate system for the subsequent operator analysis and design, rather than to constitute theoretical novelty on its own. The logic is as follows: first, additive aggregation sums the content and geometry terms separately and therefore has a structural blindness to binding swaps; second, scalar, channel, and matrix gating correspond to different strengths of content–geometry coupling; and finally, the full bilinear operator has a prohibitive parameter cost, so Fold-Conv combines KK shared basis matrices using geometry-generated coefficients to implement matrix gating at controlled cost.

We have rewritten the opening of Section 2 to state explicitly that outer-product factorization is a standard tensor-algebra tool and to present the “operator classification—additive counterexample—low-rank implementation” chain before the theorem. Theorem 1 has been retitled “Single-layer characterization of bilinear interactions,” and unqualified network-level statements have been removed from the abstract, Introduction, Figure 1, and contribution list. The theory therefore explains the design motivation and inductive bias rather than using a standard algebraic identity to claim additional mathematical novelty.

AuthorsOpenReview ↗

Response to Q3: Bias–Variance and Limited Data

We agree with the reviewer that a larger function class does not automatically improve generalization on finite training data. Hyper-Fold is not designed to increase expressivity without constraint; instead, it introduces a more appropriate inductive bias for content–geometry interactions. Geometry conditioning allows the model to directly exploit local structural relationships, the KK basis matrices shared across edges restrict the kernel's degrees of freedom, and rank-KK factorization serves both as a computational approximation and as capacity control. Thus, our observation is that the combination of structured interactions and controlled capacity is effective, not that more parameters or a larger function class is always better.

The new results provide empirical support for this interpretation. Across three runs in Table 3, Matrix outperforms Channel on all four metrics, with native Matrix sample standard deviations of 0.001–0.005. Appendix Figure 10 further shows that increasing base width does not monotonically improve performance: Channel reaches its best Pocket [email protected]/[email protected] at C1, with values of 0.611/0.455, whereas Matrix performs best at native C0, with 0.618/0.473. Matrix's EC Fmax@50 also decreases from 0.780 at C1 to 0.772 at C2 before recovering to 0.776 at C3, rather than increasing continuously with parameter count.

These results support the interpretation that inductive bias and capacity control both matter under limited data, but they are not a general bias–variance theory and do not guarantee optimality at every data scale. We have added this discussion to the revised appendix under “Repeated-run and capacity-scaling analysis,” clarifying that geometry conditioning provides a structured inductive bias, while shared basis matrices and rank-KK factorization restrict kernel degrees of freedom and control capacity. The added text is marked in blue in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.