- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
- The paper introduces are large RNA-MD dataset, more than 2500 100ns RNA MD trajectories, including different very useful splits based on structure and flexibility.
- One application for such a dataset is training MD surrogates, i.e. ML models generating trajectories that supposed can stand in for the MD. The authors validate this with a bond MAE, O3P max errors, chiral flips, clashes, an RMSF correlation coefficient and some PCA-W2 metric. The later two I don't understand, and their explanations, even though crucial for the dataset being suited as a benchmark are hidden in the appendix.
- The authors train a model in a multi-task fashion, that beneath this MD surrogate function learns two supervised tasks from single conformers: residue covariance and motion coupling between residues. Mysteriously, this is called physical grounding.
- Without any motivation, the authors restrict the training of the model to 12h, and apply the same constraint to all baselines. No analysis of the computational performance, the degree of optimization or other influential factors is given as part of a discussion in what sense this is a fair cutoff.
- The model is compared to three other models regarding the previously mentioned MD surrogate metrics, where the model beats alternatives.
- The model furthermore is compared to five models on predicting MD trajectory properties from single frames. Here I assume that the references are trained in a supervised fashion. Their model is better
- In a third evaluation, the resulting representations are used for the downstream of RNA ligand binding prediction, where also the trained model beats the baselines in several cases, in 2/8 cases even by a fair margin.
Critical strengths
- The paper provides the largest, most diverse MD dataset for RNA that I am aware of. This in itself is an invaluable resource to the community. I have criticism (see below) but still this is clearly worth publishing.
- The paper sketches a way to build a powerful model on the basis of the dataset by means of multi-task learning. Such a MD surrogate is challenging to build, especially as even with the dataset, data is still scarce. The proposed multi-task learning approach seems to point in a good direction, though I don't think we see a solution.
Critical weaknesses
- I have trouble understanding why RNADynBench is a benchmark. The dataset is a great contribution to the scarce world of RNA data, but where is are the real metrics with physical meaning? The considered metrics do not study dynamics at all, but only ensemble averages. They only look at the atom-level but not on the vastly important secondary structure not even speaking of the fold. In my opinion, the metrics are what makes a benchmark a benchmark. For example, the authors could have picked individual RNA molecules, split their conformational space into two regions and check the relative occupation in a trajectory - and hence measure the free energy difference.
- RNADynNet uses a complex multi-task learning paradigm that I found harder to grasp than I expected. The alignment loss term I did not understand at all. While it is plausible that there is a certain amount of mutual benefit for the different learning tasks, I struggle with the situation that the only presented model is so complex. Why not work with a simpler baseline first and do hyperparameter optimization. Why such an unmotivated cap in compute?
The provided dataset is a clear step forward, but here the authors rightfully point out that 100ns are too short. In RNA, single temperature runs of even 10x longer are to short for molecules to explore conformation space. Unfortunately, the highly reduces the value of the dataset for the community and even poses the risks of non-expert uncritically relying on it, not knowing that it is not equilibrated. (this is the justification for my soundness rating).
Questions for authors
- Why did you decide for the 12h cap in compute?
- Why did you not provide longer trajectories, at least for a subset of the molecules?
- How did you optimize hyperparameters of your models and your baselines.
8 comments
W3+Q2: 100-ns trajectories are too short and not equilibrated. Why not provide longer trajectories for a subset of the molecules?
With RNADynBench, we target near-native, short-timescale dynamics around experimental structures, the regime in which trajectory surrogates and single-conformer dynamics predictors are trained and evaluated. These trajectories sample local fluctuations, correlated motions, and base-pair opening around each experimental structure, though not the full conformational distribution. We prioritized structural diversity: the release covers 1,469 RNAs (258.5 s), more than existing RNA trajectory resources (Table 1). Tenfold longer runs for even 10% of the systems would roughly double its cost. RNA conformational exchange can occur on microsecond-to-millisecond timescales [1], and converged sampling remains challenging even for small RNA motifs [2]. Equilibrium sampling of RNA is therefore an open research problem that a tenfold longer run would not solve, and we do not consider it a criterion that a dataset of 1,469 RNAs can currently meet.
Within this scope, we ran a block-averaging analysis [3] on all 2,585 trajectories (new Appendix A.4): 94% of the trajectories with canonical base pairs contain at least 20 uncorrelated samples of base-pair retention (Table B). Base pairing is therefore well sampled within 100 ns. C1' fluctuations, which underlie our RMSF metric, pass the same test in 65% of trajectories, and the radius of gyration in 46%.
Longer trajectories (Q2). We extended trajectories to 500 ns for three systems (two with low and one with high flexibility). We compare each later 100 ns window with 0-100 ns and use the two halves of 0-100 ns as a control (Table C). Every later frame lies within 2.4 Å C1' RMSD of a frame from 0-100 ns. On these three systems, replacing the 100-ns reference with the 500-ns reference changes the RMSF r of BioKinema, MDGen, and RNADynNet by at most 0.04 and leaves their ranking unchanged.
We state this scope in the revised Limitations paragraph and in the dataset card. We will release the extended trajectories and per-trajectory sampling diagnostics with the dataset.
Table B (Table 7). Block-averaging analysis after discarding the first 30 ns.
| Observable | Trajectories with uncorrelated samples |
|---|---|
| Base-pair retention (trajectories with canonical pairs) | 93.9% (2055/2188) |
| C1' mean-square fluctuation | 64.7% (1673/2585) |
| RMSD to trajectory average | 60.9% (1574/2585) |
| Rg | 46.3% (1198/2585) |
Table C (Table 8). 500-ns extensions (ranges over the four later windows).
| System | RMSF r, later windows vs. 0-100 ns | RMSF r, first vs. second half | Base-pair occupancy MAE, later windows | Occupancy MAE, first vs. second half | Max. C1' RMSD to nearest 0-100 ns frame |
|---|---|---|---|---|---|
| 1F27 | 0.67-0.72 | 0.73 | 0.03-0.04 | 0.06 | 1.6 |
| 1L3D | 0.83-0.97 | 0.97 | 0.02-0.07 | 0.07 | 1.8 |
| 402D | 0.93-0.97 | 0.94 | 0.02-0.07 | 0.02 | 2.4 |
[1] Invisible RNA state dynamically couples distant motifs. PNAS, 2014.
[2] Toward Convergence in Folding Simulations of RNA Tetraloops. JCTC, 2022.
[3] Quantifying uncertainty and sampling quality in biomolecular simulations. Annual Reports in Computational Chemistry, 2009.
Q1:Why did you decide for the 12h cap in compute?
We would like to clarify that 12 hours was an approximate training budget and not a cap on every method. On two H200 GPUs, RNADynNet trains in about 9 hours and BioKinema in about 12.6 hours. We only cut off ConfRover at 12 hours because its official schedule takes over 120 hours. We have since trained it for 48 hours, selected its checkpoint by validation loss, and report this model in the revised Table 2. Compared with the 12h model, some metrics improve while others degrade, and it remains worse than RNADynNet on most metrics.
Q3: How did you optimize hyperparameters of your models and your baselines.
We use one policy for all methods: published settings where a single training run takes 9 hours or more (see Q1), and a grid search where training is cheap. We therefore retrain the baselines on RNADynBench with their released hyperparameters and take most base settings of RNADynNet from BioKinema. We set the maximum alignment weight (20) by gradient-ratio calibration, so that its gradient contribution is comparable to that of the generation objective, and we set the covariance and motion-coupling weights to 1, following BackFlip. We use these settings in all ablations. For the frozen probes and simple descriptor baselines in Table 3, we search the same grid of six learning-rate and weight-decay settings.
We thank the reviewer again for the time and detailed feedback. We believe that the new metrics and analyses, the 500ns extensions, and the clarified method description address the reviewer's concerns, and we kindly ask the reviewer to reconsider the assessment in light of the revised paper. We are fully available for discussion should any concerns remain.
Thank you for the extensive replies. I will consider your points thoroughly.
We sincerely thank the reviewer for the detailed and insightful review. We are encouraged that the reviewer regards RNADynBench as the largest, most diverse MD dataset for RNA and an invaluable resource to the community. We address each point below.
The revised Section 4.1 defines both metrics, and Appendix C.4 gives the full definitions and explanation. RMSF r is the Pearson correlation between the per-residue C1' fluctuation profiles of generated and reference trajectories. It measures agreement in the relative fluctuation pattern across residues. PCA-W2 is the 2-Wasserstein distance between the distributions of generated and reference frames in the plane of the first two principal components of reference MD.
We supervise intermediate representations with MD-derived residue covariance and inter-residue motion coupling. Both are physical observables of the MD simulations, which is why we called this supervision physical grounding (PG). Because both targets describe fluctuations and correlated motions, we now use the more specific name dynamics grounding (DG) in the revised version.
Our model builds on BioKinema and follows its evaluation protocol for comparability, replacing protein-specific metrics with measures of O3'-P linkage geometry and sugar chirality. In the revised Table 2, we add metrics for secondary structure and dynamics. Base-pair occupancy error (Pair occ. MAE) compares how often each residue pair is base-paired in generated and MD trajectories. Lag-dependent RMSD measures the displacement between frames 1 to 20 ns apart, so unlike an ensemble average it depends on frame order. We report its error against MD (LAG-RMSD MAE) and its correlation with MD across RNAs (LAG-RMSD r). Backbone continuity (O3P cont.) is the percentage of generated trajectories whose backbone stays connected in every frame. At the global level, the existing PCA-W2 metric compares the conformational distributions of generated and MD trajectories.
We thank the reviewer for this suggestion and have run the analysis (new Appendix D.4). We use the fraction of formed base pairs in a stem, Qstem, to divide conformational space into an open region (Qstem<0.5) and a closed region (Qstem≥0.5), with occupancies popen and pclosed. Occupancies are meaningful for stems that visit both regions repeatedly, so we analyze the 38 stems, one per target, that open and close at least twice in the reference MD. The apparent free-energy difference is ΔG/kBT=−ln(pclosed/popen). Compared with BioKinema, RNADynNet has lower apparent free-energy error, transition counts closer to MD, and higher region-occupancy error.
Table A. Two-region analysis of stem opening and closing.
The revised Section 3 now thoroughly describes the architecture before the losses. In summary, our architecture is a single backbone trained with three objectives. With the generation objective, we train it to denoise MD trajectory windows. With alignment, we match the features the backbone computes from a single frame to those it computes from the trajectory window containing that frame. This is self-distillation: the trajectory pass acts as teacher and the single-frame pass as student, with no separate teacher network, and it transfers trajectory information to the single-frame features. With dynamics grounding (DG), we supervise these single-frame features with MD-derived residue covariance and motion coupling. Alignment and DG add three prediction heads and one truncated single-frame forward pass per training step. At inference, trajectory generation and descriptor prediction each take a single conformer as input.
We verify each component by ablation. Adding alignment to generation-only training raises the frozen-probe RMSF r on test_flex from 0.62 to 0.69, and adding DG raises it to 0.78 and improves all seven descriptor metrics on both test sets (Table 3). Adding DG also raises trajectory-level backbone continuity on both test sets (Table 2). As the simpler baseline the reviewer asks for, we trained the same backbone with descriptor supervision alone (DG-only, Table 3): RNADynNet obtains better performance on five of seven metrics on each test set.