acceptodds
Withdrawn from ICLR 2027

Biology-Guided Synergistic Learning of RNA Foundation Models for Secondary Structure Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

RNA secondary structure prediction aims to identify base-pairing patterns from primary sequences and requires rich biological priors for accurate modeling. Recent foundation models trained with large-scale pretraining encode rich biological priors into high-dimensional representations, yet their effective exploitation for downstream folding remains challenging. To address this issue, we revisit RNA secondary structure prediction from a representation learning perspective, focusing on a more effective exploitation of existing prior-aware representations rather than developing new deep learning modeling paradigms. We propose BioSynFold, a biology-guided synergistic learning framework for RNA secondary structure prediction that leverages foundation models from both model and data perspectives. From the model perspective, we propose a unified framework that integrates multiple RNA foundation models and introduces a progressive integration strategy to better leverage heterogeneous representations. From the data perspective, we exploit weak signals derived from sequence-level statistical and compositional properties to guide a mixture-of-experts routing mechanism, enabling condition-aware specialization and improving representation quality. Extensive experiments on multiple datasets indicate that BioSynFold significantly outperforms the state-of-the-art methods. Our code is available at https://anonymous.4open.science/r/BioSynFold.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
secondary structure prediction, RNA foundation model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ziBcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes BioSynFold for RNA secondary structure prediction. It extracts frozen nucleotide embeddings from RNA-MSM, RiNALMo, and ERNIE-RNA, projects them into a common space, and combines them using learned weights. A sequence-level mixture of CNN experts is routed using sequence length, nucleotide composition, and 2-mer frequencies. The resulting representation is decoded using the existing sincFold prediction head. Experiments cover RNAStrAlign, ArchiveII, bpRNA-TS0, long-range contacts, and cross-family evaluation on bpRNA-new. The paper reports improvements over the selected baselines, together with encoder-combination, fusion, routing, and computational-cost analyses.

Critical strengths

The gains extend to unseen families. Tables 1–2 report F1 scores of 0.753 on bpRNA-TS0 and 0.663 on bpRNA-new, versus 0.737 and 0.639 for the strongest listed competitors in those settings. Table 5 reports a bpRNA-new mean of 0.659 ± 0.003 across six seeds. These results, along with the long-range-contact evaluation, are more informative than the nearly saturated RNAStrAlign result.

The fusion, encoder-combination, and routing ablations help assess the architecture (Tables 3, 4, and 8). In particular, the capacity-adjusted fusion baselines and shuffled-descriptor control allow a closer look at where the improvements arise.

Critical weaknesses

The method combines several established components, while the evidence for the proposed biological routing mechanism is weak. It uses three frozen encoders, normalization and linear projections, global softmax fusion weights, a sparse MoE, and the sincFold decoder (Sections 3.2–3.5). Equation (2) learns a shared fusion-weight vector rather than selecting encoders for each input. The best two-encoder combinations reach 0.751 and 0.750 F1, compared with 0.753 for all three (Table 4). Shuffled routing descriptors reach 0.747 versus 0.753 with the intended descriptors (Table 8), without repeated-seed uncertainty for that comparison. Family-dependent expert usage is interesting, but does not by itself show that experts capture complementary structural mechanisms. I would need stronger evidence that the specific fusion and routing choices add more than extra representations and predictor capacity.

The accuracy–cost tradeoff is not sufficiently established. The comparison does include PriFold and DEPFold from 2025, plus RFold and sincFold from 2024. However, the cited DSRNAFold and RNADiffFold methods are absent, PriFold is omitted from Table 2, and there is no well-tuned or fine-tuned single-foundation-model predictor under matched supervision. Appendix C separates embedding extraction from cached prediction. For a previously unseen RNA, the three extraction averages sum to 69.6 ms/RNA, followed by 2.78 ms/RNA for prediction. Their sum, about 72.4 ms/RNA, is only a sequential estimate from separate timings, not a measured end-to-end latency; sincFold's reported prediction time is 2.16 ms/RNA. Cached embeddings also total about 1.67 MB/RNA. Given the 0.002 F1 gain over the best two-encoder configuration, the complete cost of the third encoder, including any RNA-MSM input preparation, needs to be justified.

Questions for authors

Do the 0.006 F1 gain over shuffled descriptors and the 0.002 gain over the best two-encoder configuration persist across seeds? Can capacity-matched controls isolate the contribution of the biological descriptors?

How does BioSynFold compare with a tuned or fine-tuned single-foundation-model predictor, DSRNAFold, and RNADiffFold under matched splits and supervision? Why is PriFold absent from the challenging evaluations?

What is the measured sequence-to-structure time for an uncached RNA across different lengths, including encoder passes, input preparation, cache I/O, and decoding? Does RNA-MSM receive an MSA or a single sequence, and is any MSA construction included? Please also report the best two-encoder system on the same basis.

Reviewer_JFKCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I think the paper presents an interesting approach into RNA secondary structure prediction. Comparing with previous algorithm-based or supervised-learning-based methods, this paper merges the feature embeddings of RNA foundation models to do the task, which aligns with recent advances in LLMs. More detailed into the method, the paper integrates representations from 3 pretrained RNA foundation models through feature alignment and learnable weighted fusion. The authors further introduce a MoE module, which uses sequence-level descriptors to guide expert routing. The experiments are done on the widely-used benchmarks for RNA secondary structure prediction (bpRNA-TS0, etc), and the methods showed competitive performance comparing to baselines under multiple tasks. In all, I think the paper addresses a practically relevant problem and reports promising empirical results. But I do have several questions(concerns) regarding the significance of the method, the performance of the experiments and the evidence of the claims.

Critical strengths
  1. I think the paper investigates a meaningful direction in RNA secondary structure prediction. Given that different RNA foundation models are trained with different objectives and data sources, exploring how to combine their representations is reasonable. The introduction of MoE and biologically guided routing is also an interesting attempt to better utilize pretrained representations for downstream RNA tasks.

  2. The paper reports encouraging performance across several RNA secondary structure prediction benchmarks, particularly in long-range base-pair prediction and cross-family generalization. The authors also provide ablations of different foundation model combinations, fusion strategies, and expert routing mechanisms. I think these experiments provide useful insights into the proposed framework. The overall presentation is also clear and well organized.

Critical weaknesses
  1. I am not fully convinced that the proposed architecture sufficiently supports the claimed synergistic learning between RNA foundation models. The fusion mechanism uses globally learned weights to combine different representations, while the MoE operates on the already fused features rather than explicitly modeling interactions between individual foundation models. Therefore, it remains unclear whether the framework truly exploits model-specific complementary information or mainly benefits from combining more pretrained features. In addition, the routing descriptors, such as sequence length and nucleotide composition, may be strongly correlated with RNA family identity. I think further analysis would be helpful to distinguish meaningful structural specialization from family-specific statistical patterns.

  2. Although the paper reports strong results, I think the experimental comparisons could be more comprehensive. In particular, the paper does not directly compare against relevant RNA foundation model-based prediction pipelines, such as fine-tuned RiNALMo and the unified RNA language model benchmarking framework discussed in related work. The current ablations do not fully replace these comparisons. I think evaluating these methods under consistent data splits and metrics would help establish the advantages of the proposed approach. It would also be helpful to clarify whether RNA-MSM requires actual multiple sequence alignments during inference, since the paper mainly describes a single-sequence prediction setting.

Questions for authors
  1. Could the authors further clarify how the proposed framework achieves synergistic learning beyond standard multi-model representation fusion, particularly given the globally learned fusion weights?

  2. Could the authors provide further evidence that the MoE learns meaningful structural specialization rather than mainly distinguishing RNA families through sequence statistics? I am particularly interested in its behavior on unseen RNA families.

  3. Could the authors clarify whether RNA-MSM uses actual MSAs or only individual RNA sequences during embedding extraction? Also, would it be possible to compare with existing fine-tuned RNA foundation model-based secondary structure prediction methods under consistent evaluation settings?

Reviewer_VWm6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a new method for RNA secondary structure prediction, BioSynFold, based on an adaptor that combines RNA-FM embeddings into a joint space, and a MoE routing mechanism guided by simple RNA sequence features like length, nucleotide distribution, and k-mer distributions. The authors evaluate BioSynFold across different datasets and provide several ablation studies for the MoE and the combination of FM embeddings. Overall, the most interesting part of the paper is the combination of FM embeddings, which is well motivated with a representation learning view. The paper is well written and motivated.

Critical strengths
  1. I think the main strength of the manuscript is the combination of different FM representations into a joint space, which seems to strongly contribute to the performance of BioSynFold. The combintation is well motivated by different biological features that might be learned during FM training. The ablation studies (specifically Fig 4 and Table 4 (App.)) strongly support the gains of this approach. I'm actually currently unsure if it would be a sufficient contribution if the authors could somehow show that these FM representations actually really carry different biological information. I think that would be much more interesting than another small gain on mainly redundant benchmark datasets (see weaknesses).
  2. The routing approach via MoE is also interesting, and the framework supports inclusion of additional FMs, as well as different predictors (see also questions).
Critical weaknesses
  1. The evaluation is weak. The RNAStralign dataset is highly redundant and was already shown to share families with ArchiveII which limits the transfer argument [1]. Similarly, the BpRNA base set was shown to be mainly comprised of four RNA families [2]. All three sets are not well suited for evaluation of DL methods if not carefully split for homology (or sequence and structure similarity) before evaluation, which is not the case here. Furthermore, at least ArchiveII contains canonical base pairs only, and I'm currently unsure whether BioSynFold predicts non-canonical base pairs at all (see questions). The most interesting dataset used in the evaluation is the BpRNA-new dataset, however, this set also contains canonical base pairs only (and I do not understand how the training data was produced as the augmentation procedure cites the ViennaRNA package 2.0 publication which I don't think provides any augmentation procedure; I also could not find any when looking into the paper again).
  2. (Also somewhat related to 1.) I think the authors should be more clear in the limitations section about what the comparison shows. Some earlier methods (e.g. SPOT-RNA; and if I remember correctly, also UFold depending on which checkpoint was used during evaluation) predict all kinds of nucleotide interactions, including higher-order interactions which is a clear gain over classical predictors (e.g. dynamic programming approaches that already struggle with pseudoknot predictions) but might decrease performance on datasets that do not contain such pairs. For BioSynFold, the interactions are restricted to pair interactions during post-processing as far as I understood. Overall, the evaluation therefore does not really allow to assess BioSynFold performance.

[1] Qiu, Xiangyun. "Sequence similarity governs generalizability of de novo deep learning models for RNA secondary structure prediction." PLOS Computational Biology 19.4 (2023): e1011047. [2] Flamm, Christoph, et al. "Caveats to deep learning approaches to RNA secondary structure prediction." Frontiers in bioinformatics 2 (2022): 835422.

Questions for authors
  1. Which base pairs does BioSynFold actually predict? The binary interaction map representation can express all kinds of interactions, but the post processing limits this to pair-wise interactions, or? Do these contain non-canonical base pairs?
  2. In how far does this depend on the chosen downstream predictor?
  3. Did the authors try different FMs not shown in the paper? How did it work? If the story of different biological features holds, shouldn't FMs trained on similar data/with similar strategies add similar information to the joint representation?
  4. Did the authors use any other predictor than sincFold?
  5. Why is the BpRNA-TS0 dataset a large scale benchmark? Isn't that the one set with ~1300 samples up to length 500 while ArchiveII includes ~3900 samples or so including larger RNAs? Similarly, isn't RNAStralign also larger?
  6. In Figure 5, the authors show the utilization of experts across RNA families. Would it be possible to track family knowledge of the representation only by looking at expert utilization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.