acceptodds
Under review as a conference paper at ICLR 2027

Temporal Triangular Attention for Protein Molecular Dynamics

Abstract

Protein dynamics carry critical information about protein function, yet current machine learning methods offer few ways to encode and reason over them. Molecular dynamics (MD) trajectories provide a rich view of these dynamics that is complementary to the sequence and evolutionary signals exploited by protein language models, but transforming them into effective protein representations remains an open challenge. In this work, we introduce Temporal Triangular Attention (TTA), which extends triangular attention from a single static structure to a temporally evolving sequence of structures. For each residue pair, TTA jointly selects a mediating residue and a trajectory frame, enabling pair-conditioned reasoning over when and where relevant motion occurs. We instantiate TTA in ProTTA, an Evoformer-style MD encoder that incorporates residue trajectories into pairwise structural embeddings. By pretraining ProTTA to forecast the dynamic cross-correlation of the next trajectory window, we obtain representations of coordinated motion without any functional annotations. We evaluate ProTTA on two benchmarks spanning function prediction and dynamics forecasting. For function prediction, ProTTA outperforms existing MD-based encoders and, when combined with sequence information, achieves the best result on 22 of 24 metrics against both sequence models and MD encoders. For dynamics forecasting, ProTTA outperforms the MD-only baseline and performs comparably to a sequence-conditioned model while using dynamics alone.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Protein Analysis, Molecular Dynamics, Triangular Attention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ygUEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce temporal triangular attention, inspired by the triangular attention used in Alphafold-style models. This is in particular to the application for molecular dynamics simulations where currently deep learning models compress the trajectory into a single embedding and do not capture the residue-pair representations. The authors incorporate the temporal triangular attention into ProTTA, an Evoformer-style encoder, where it replaces the triangular attention. ProTTA is evaluated on mdCATH and ATLAS molecular dynamics datasets for dynamics forecasting and function prediction results.

Critical strengths
  • The paper is well written, easy to understand and follow with a clear schematic describing the architecture.

  • The development of the temporal triangular attention for molecular dynamics accounting for the time frame as well is interesting and novel.

  • Experiments across mdCATH and ATLAS which are popular MD datasets show the robustness of the model, and competitive performance against the baselines make the results stronger.

Critical weaknesses
  • The metrics for forecasting report JSD, recall, F1, and RMSD. However, the physical and structural validity of the generated molecules is not checked. These could be done using metrics such as Ramachandran outlier rates and clash scores which would show whether the forecasted conformations are physically plausible.

  • Additionally, from Table 4, it looks like ConfRover outperforms ProTTA on every reported metric (JSD, recall, F1, and RMSD) which overstates ProTTA’s results.

Questions for authors
  • Have the authors considered structural validity metrics such as Ramachandran outlier rates and clash scores?

  • Have the authors considered other tasks such as ensemble generation or energy-based path generation? It could be interesting to look at considering the emphasis on a good representation for molecular dynamics trajectories.

2 comments

Thanks for the insightful comments, and we kindly address them as following:

Comment: Physical and structural validity of forecasted conformations.

We agree that distributional metrics alone do not establish physical plausibility. In the current work, ProTTA operates on Cα coordinates only. Both the trace embedding and the forecaster represent each residue by its Cα atom, and the training objective (EDM denoising of Cα displacements, plus a Cα–Cα bond-length term) is defined at that level. Backbone-level metrics such as Ramachandran statistics require backbone dihedrals, which a Cα-only model does not produce. However, these additional properties can be added incrementally. We also note that many protein structure generation models operate on the pair representation, which is the main feature in our architecture, thus extending to all atom structure models can be achieved. We therefore see this work as a deliberate first, incremental step. It provides evidence, at the Cα level, that extending the triangle mechanism across time helps model molecular dynamics, both for function prediction and for forecasting.

Comment : Comparison to ConfRover

We acknowledge that ConfRover outperforms ProTTA, and we did not intend to claim otherwise. Our claim, in lines 86–88, is that “ProTTA reduces forecasting RMSD by 20% relative to MD-only baselines and comes within reach of sequence-conditioned ones without access to sequence.” We position ProTTA as an MD embedding model. Designing a way to incorporate sequence into dynamics forecasting is non-trivial and well beyond our scope, so we keep ProTTA MD-only for this task.

Comment: Ensemble generation and energy-based path generation.

Ensemble generation is the similar of the two to our current setup. Our forecasting evaluation already treats rollouts as ensembles: it reports JSD between coordinate distributions, and our released evaluation code also computes pairwise-RMSD distributions and Wasserstein distances. Our multi-start protocol also samples many rollouts per protein.

The other tasks, like energy-based path generation, mentioned by the reviewer require a different generative objective and evaluation, and we list it as future work. We appreciate the suggestion and will pursue it in follow-up work.

Thank you for the responses. I maintain my positive score. Best of luck!

Reviewer_bWW7OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • The paper addresses the question of how to embed a sequence of MD frames of a protein into a representation suited for downstream tasks.
  • The authors introduce temporal triangular attention, a variant of triangular attention that is used in Alphafold to update the pair representation. Here, each pair ij is update based on the edges of the triangle ijk where k iterates over all other residues.
  • I have severe problems understanding the suggested temporal triangular attention. The authors introduce the notion of legs that don't resonate with my imaginations. The authors jump from embedding motion vectors to pairs, and then I'm lost. Also the visualization in Fig 2 is not very helpful. It has very small font size, and the caption is not very informative. Also is uses jargon ("searches" - I am almost sure there is not search involved, but some form of attention).
  • With TTA as key building blocks, the authors construct ProTTA. This model is pre-trained on a cross-correlation of motion in a supervised way.
  • Then, the model is fine-tuned for function prediction and MD forecasting.
  • In the function prediction benchmarks, ESM2 sequenced embeddings are fused with the model output. As model, the trained ProTTA is compared to three other MD embedding models. All benchmarks check how big is the gain over pure ESM2 embedding input.
  • On the function benchmarks, we see a slight increase in scores by the model. The authors claim this is SOTA but to my knowledge, specialized models outperform the models by a fair margin. Seen from the perspective of "what is the most informative embedding" the trained model sets a new SOTA. But here it is not clear if this is only a result of TTA, or what else might be a reason.
  • The model can be fine-tuned as an MD surrogate performing similar to ConfRover and mdgen.
Critical strengths
  • The paper makes an important contribution in the field of processing MD trajectories of proteins with ML models. It devises a new architecture from the Triangle attention mechanism that even though I don't quite understand it plausibly provides an innovation.
  • The proposed model outperforms other MD trajectory embedding methods. The functional annotation scores clearly benefit from the trajectory embedding more than from the other embeddings.
Critical weaknesses
  • The presentation lacks clarity. During the review, I have given up understanding TTA. It might be that I did not allocate enough time, but still this is a bad sign for a paper that specifically presents a new architecture. The explaining text is hard to follow, the explaining graphics not very helpful and overall I'm lost.
  • The results section is written with not enough care. The function benchmarks are not explained very well and impossible to understand without external reference. The adaptation procedure is not well explained and even hidden in C2 (labeled ProTTA, even though the same procedure is applied to the embeddings of the other models, too). It is not mentioned if this is a full fine-tuning or uses frozen embeddings only (I guess so). Why are do some results even get worse when adding MD embeddings?

Overall, despite encouraging benchmarks, the lack of clarity and precision in presentation gives me no chance but to vote for a rejection.

Questions for authors
  • Can you please explain TTA more clearly?
  • Can you please add the ESM2 baseline to table 5?
3 comments

We thank the reviewer for this candid comment. We agree that if a careful reader cannot follow the core operator, our presentation needs to be improved. Below we tried to provide a brief description to facilitate the reviewer to carefully evaluate the manuscript. In addition, we plan to provide a revised Sec. 3.2, Sec. 4 and the appendix with sufficient information that readers are able to follow.

Comment : TTA is hard to understand; the figures do not help.

Temporal Triangle Attention (TTA) extends on the well examined triangle mechanism that operates on spatial dimensions and is widely used by Alphafold like architecture to include temporal dimension. TTA operates on a feature tensor of a residue pair (i, j) by looking at a nearby third residue k at other time points or frames. Each candidate (k, frame u) is scored twice: how well k’s motion at u agrees with i’s motion at frame t, and how well it agrees with j’s (Eq. 2). The two scores are combined so that k must match both ends of the pair (Eq. 3); this is what makes the update a triangle. A single softmax over all (k, u) then selects which residue and when (Eq. 4). The pair feature z_ij is updated with the motion of the selected residues, averaged over the window (Eqs. 5–6).

The “legs” are scoring functions that relate the starting point i to each trace k, and each trace k to the endpoint j. Following triangular attention, which decomposes the i-k-j relationship into i-k and k-j, we introduce these two “legs” to measure each part. The motion information is stored in a pairwise representation matrix that encodes the i-j relationships, and the i-k and k-j scores described above are used to update that pairwise representation.

We are happy to provide further details or explanation. It would be very helpful if the reviewer could point out which parts, beyond the paragraph mentioned in the summary, were difficult to follow.

Comment : The function benchmarks are not explained.

We had provided the brief details of function-prediction labeling details in Appendix C.1 and the model architecture in Appendix C.2. For additional clarity, below is a description for reviewer and reader who may be unfamiliar with this area. We plan to add this to the revised version.

GO prediction. Each protein is labelled with Gene Ontology terms that have experimental evidence in UniProt, propagated to all ancestor terms. We keep terms with ≥ 50 training positives: 58 / 173 / 59 for MF / BP / CC on mdCATH, and 12 / 56 / 23 on ATLAS (Table 7). This is a multi-label task: one model predicts all terms. Fmax is the protein-centric F-score at the best single decision threshold (CAFA protocol). AUPR is the area under the precision–recall curve, averaged over terms. HPA classes. Four coarse, non-exclusive classes (enzyme, metal binding, membrane protein, transporter), derived from UniProt keywords (Table 6). We report [macro-]F1 at threshold 0.5, and AUPRC and AUROC averaged over the four classes. Splits are cluster-disjoint: proteins are clustered at 30 % sequence identity and whole clusters are assigned to train, validation or test, so no close homolog of a test protein is seen in training.

We are not sure what the reviewer means by “hidden in C2”, and we would be happy to address any specific missing or confusing points. Perhaps, the reviewer could provide additional details for us to address this issue.

Comment: Why do some results get worse when MD embeddings are added?

As discussed in Section 2, many of these MD embedding methods average out the motion and therefore cannot produce embeddings that are useful for function prediction. Adding them to the ESM-2 embedding consequently does not improve the results.

Comment: Adding ESM to Table 5:

(This response is the same as our response to Reviewer 8Dso, since both reviewers raised the same question.)

We now provide ProTTA + ESM-2 dynamics forecasting results, using the same fusion mechanism as in function prediction. We would also like to note that, as stated in line 353, “adapting them (MD embedding methods) to generate per-residue coordinates, unlike predicting protein-level labels, is non-trivial”. Specifically, while predicting protein-level labels with sequence-only models such as ESM-2 has been widely studied, dynamics forecasting based on ESM-2 has not been explored. Any underperformance therefore cannot be attributed specifically to ProTTA, to ESM-2, or to the lack of a suitable fusion technique. Our focus is on developing an MD embedding method, so adapting ESM-2 to dynamics forecasting is out of scope.

JSDRecallF1RMSD
MDGen.569.293.4341.09
ConfRover.51.42.58-
ProTTA + ESM-2.582.291.4341.33
Area_Chair_7rsrOpenReview ↗

Please help to join the discussion

Dear Reviewer bWW7,

The discussion deadline is approaching. Please help to join the discussion and offer the reviewer's opportunity to further respond to your concerns.

Thanks!

AC

Thank you for addressing my concerns. This has added clarity and I'm willing to raise my vote to 2. I would have liked to see an updated version of the manuscript with marked changed, but I also don't think this would have affected my rating.

Reviewer_8DsoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Temporal Triangular Attention (TTA), which extends the triangular attention from a single static structure to a trajectory window: for each pair of residues (i, j) in frame t, the method selects an intermediate residue k from a frame u. Based on this, the authors built an Evoformer-like encoder for molecular dynamics (MD) trajectories, ProTTA, and performed self-supervised pretraining by predicting Dynamic Cross-Correlation (DCC). The model achieved good performance on the tasks of function prediction and dynamic forecasting.

Critical strengths
  • The core idea is straightforward. Considering two aspects: first, the limitation that triangular attention can only process a single static structure; and second, the fact that MD trajectories contain a large number of conformational frames, which provide more information than a single static structure. This naturally led to extending the selection of intermediate residue k to other frames.

  • The model achieved good results on both function prediction and dynamic forecasting

Critical weaknesses

The writing is quite unclear. The beginning of the paper gives the impression that it proposes a method for obtaining a representation for the entire MD trajectory, but in the experiments, the method can actually only handle a small window (only 8 frames). Still writing issues: “TTA extends this principle to Minkowski space, which allows the model to naturally capture structural motions across conformations (see Appendix B for theoretical support). As a result, TTA is beneficial not only for the protein-level objectives that motivated its design, but also for residue-level tasks such as structure forecasting, as we will demonstrate empirically in Section 4.4.”  The paper does not use the Minkowski metric, so the statement “extends this principle to Minkowski space” is an exaggeration. Appendix B only shows the triangle inequality  for 3 points in Euclidean space. And the causal relationship in the second sentence, “As a result,” is unclear. As mentioned in 1, TTA has high computational complexity and is difficult to scale. In practice, it can only be adapted to small windows. The evaluation is not complete. While ProTTA + ESM-2 got the best performance in the function prediction task, there are no results for ProTTA + ESM-2 in the dynamics forecasting task. It is unclear whether the model actually utilizes cross-frame temporal information. In the experiments, the trajectories were heavily downsampled: for each trajectory in mdCATH/ATLAS, only 64 frames were selected, and then an 8-frame window was chosen from those 64 frames. This resulted in the fact that neighboring frames within the window might be hundreds of frames away from each other in the original trajectory, which is more similar to sampling. It would be best to add control experiments with shuffled frame order. The benefits of pre-training are very slight, so its contribution is unclear.

Questions for authors
  • clarify performances on longer trajectories
  • (weakness 4) add results for ProTTA + ESM-2 in the dynamics forecasting task
  • clarify weakness 5
  • clarify whether pre-training is really impactful
  • where is Minkowski metric ever used ? If not, simply remove
2 comments

Thanks for sharing the comments, and we apologize for the confusion and for the unclear descriptions of our configuration, which may have made TTA harder to understand. We plan to provide additional descriptions of the architecture, specific components and the motivation for each of the components to the revised manuscript based on the text provided below.

Comment : The paper suggests a whole-trajectory representation, but ProTTA processes an 8-frame window.

ProTTA is a window encoder but produces a representation of the whole trajectory. In particular, We use an 8-frame window as an adaptation to computational constraints, as stated in lines 212–245. ProTTA still produces a representation of the whole trajectory: given a trajectory, we first downsample it to 64 frames, obtain 8 embeddings by applying TTA to each 8-frame window, and pass them to a readout module that aggregates the embeddings of all windows by mean and standard-deviation pooling. Since the 64 frames are sampled across the full trajectory, we expect the resulting representation to retain information about the whole trajectory. In addition, the ablations that shorten the window or remove the temporal computation clearly show that ProTTA captures cross-structure motion. We would like to emphasize that both the downsampling to 64 frames and the 8-frame window are configurations chosen in response to limited computational resources. Such limits arise naturally in MD embedding, since MD data is extremely high-dimensional and the triangular attention we build on already has cubic complexity in sequence length.

Comment : “Confusion on extending to Minkowski space”.

We would like to clarify that we never use the Minkowski metric. The claim we intended to make in Section 3.2 is that TTA extends the previous single-frame triangular attention to a form that operates across both time and space. We will correct this wording in the revision.

Comment: No ProTTA + ESM-2 result for dynamics forecasting.

ProTTA + ESM-2 for dynamics forecasting: We now provide ProTTA + ESM-2 dynamics forecasting results, using the same fusion mechanism as in function prediction. We would also like to note that, as stated in line 353, “adapting them (MD embedding methods) to generate per-residue coordinates, unlike predicting protein-level labels, is non-trivial”. Specifically, while predicting protein-level labels with sequence-only models such as ESM-2 has been widely studied, dynamics forecasting based on ESM-2 has not been explored. Any underperformance therefore cannot be attributed specifically to ProTTA, to ESM-2, or to the lack of a suitable fusion technique. Our focus is on developing an MD embedding method, so adapting ESM-2 to dynamics forecasting is out of scope.

JSDRecallF1RMSD
MDGen.569.293.4341.09
ConfRover.51.42.58-
ProTTA + ESM-2.582.291.4341.33

Comment : Does the model use cross-frame temporal information? Frames are heavily downsampled; shuffled-order control.

This downsampling is a configurable choice that balances performance against computational cost, and it can be changed. We would also like to point out that the reviewer's assumption that “neighboring frames within the window might be hundreds of frames away from each other” does not hold for either dataset. In mdCATH, trajectories are sampled with a 1 ns stride and have a median length of 400 ns, so our reduction introduces a stride of only about 5 frames. In ATLAS, trajectories are sampled with a 100 ps stride and have a uniform length of 100 ns, so our reduction results in a stride of roughly 16 frames. We further report the median RMSD between consecutive sampled frames under various sampling rates, which shows that our downsampling does not fragment the trajectory and preserves most of the motion:

Downsampled Frames3264128Full
ATLAS1.721.551.421.06
mdCATH2.201.981.781.49

Impact of pretraining: We present the pretraining procedure as an initialization technique for training our model. As shown in Table 5, most of the improvement does not come from pretraining. We report the pretraining results to support our claim that “understanding MD motion benefits function prediction”.

I acknowledge the author's response and efforts to answer my requests. I need to ponder a bit more on some of these elements. the minor aspects have clearly been answered, though.

Comments on the paper

1 comment
Area_Chair_7rsrOpenReview ↗

Reviewers, please help to join the discussion.

Dear reviewers,

The authors have uploaded the rebuttal. Please help to join the discussion and the further process.

Thanks!

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.