acceptodds
Under review as a conference paper at ICLR 2027

OptimRNA: Structure-Aware RNA Language Modeling with Factorized Pair States

Abstract

RNA function is strongly shaped by pairwise structure, yet RNA language models (RNA-LMs) rely on BERT-style Transformers that only maintain per-token representations between layers. To explicitly integrate prior knowledge of RNA pairing into sequence learning, we introduce OptimRNA, an architecture that augments the token stream with a pair state maintained and refined across Transformer layers. Initialized from canonical pairs supported by stem-compatible neighborhoods, the pair state is progressively updated from contextual token representations using signed low-rank transformations propagated by antidiagonal convolutions encoding RNA-stem patterns. Every Transformer layer updates the pair state, while later layers read it as a bounded attention bias. Its low-rank formulation enables memory-efficient pair modeling with negligible parameter overhead compared with standard RNA-LMs. We show that OptimRNA outperforms all evaluated RNA-LMs in zero-shot secondary-structure prediction and, after supervised fine-tuning, competes with substantially larger foundation models without compromising performance on tasks less directly tied to secondary structure. Controlled ablations and mechanistic analyses support the complementary roles of the pair pathway and stem-aware propagation. Together, these results establish OptimRNA as an efficient biologically-informed architecture for RNA language modeling.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
RNA Language Model, Structure prediction, RNA Foundation Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LnsoOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces OptimRNA. It is an architecture augments the token steam with a persistent pair state. It is well motivated and show strong empirical gains in secondary structure prediction. The paper also includes ablations to supported the architecture designs and analysis of the model behaviours.

Critical strengths
  1. The architecture is well motivated and easy to follow. It shows strong empirical results in secondary structure prediction and is efficient as well. The authors also provide controlled ablations to support the architecture design.
  2. The paper includes rigorous analysis of model behaviour. It provides useful insights into how structural information develops across layers and is represented in the pair and token streams.
Critical weaknesses
  1. The paper does not include comparison under matched training compute (FLOPS or GPU hours). The pair pathway increases training-step time by approximately 2.3× over the token-only backbone. But whether the architecture outperforms a backbone trained longer or with more parameters is unknown.
  2. It shows weak evidence for broad transferability. The improvements on modification prediction are modest and effects on splice-site prediction are mixed. The improvement mainly concentrated in secondary structure prediction.
Questions for authors
  1. How does performance change when models are pretrained with random symmetric or zero-initialized pair states?
  2. Could you provide comparisons for models with same training budget (larger baseline models)?
  3. What is the statistical significance of the improvements over baselines?
  4. Could you provide comparisons with AlphaFold3 on secondary structure predictions (derived from the predicted 3-d structures)?
7 comments
AuthorsOpenReview ↗

Answer to Reviewer Lnso

We thank the reviewer for their positive assessment of our work.

Q1

Thank you for this suggestion. Following your request and Reviewer j4fN's complementary proposals, we substantially expanded the architectural ablations. All variants use the same 34M backbone, pretraining data, and training recipe. We report zero-shot secondary-structure prediction AUPRC (%) on bpRNA TS0, bpRNA-new, Rivas TestSetB and RNA3DB-2D using our common nucleotide-dependency readout, without fitting a downstream predictor. We add the difference from the full OptimRNA model in parentheses, and the statistical significance from Holm-adjusted, two-sided paired Wilcoxon tests (* p<0.05, ** p<0.01, *** p<0.001).

ModelTS0 (n=1304)bpRNA-new (n=5401)Rivas TestSetB (n=430)RNA3DB-2D (n=147)
Full OptimRNA56.5945.8172.8258.37
OptimRNA with zero-prior56.13 (−0.46)47.21 (+1.41)***69.48 (−3.34)**52.03 (−6.35)**
OptimRNA without cross-layer accumulation55.63 (−0.96)*46.11 (+0.30)70.60 (−2.22)*55.27 (−3.11)
OptimRNA without antidiagonal conv53.16 (−3.43)***40.90 (−4.91)***71.10 (−1.71)***53.11 (−5.26)***
Baseline, with attention bias50.12 (−6.46)***34.05 (−11.75)***67.92 (−4.90)***50.04 (−8.33)***
Baseline, no attention bias50.50 (−6.09)***35.78 (−10.02)***65.50 (−7.31)***49.12 (−9.25)***

Zero initialization within OptimRNA does not significantly reduce performance on TS0, slightly improves it on bpRNA-new and worsens it on the Rivas TestSetB and RNA3DB-2D. It still retains a substantial advantage over the token-only backbone. Compared with the ablation of the antidiagonal convolution, the effect of removing the initial pair state is more dataset-dependent and less conclusive.

The full OptimRNA model achieves the highest average observed performance; initialization with prior knowledge contributes inconsistent and, on average, smaller improvements compared with the pair state itself and the antidiagonal convolution.

Q2

Thank you for raising the distinction between parameter efficiency and training-compute efficiency. Our controlled comparisons fix the pretraining recipe and number of gradient updates, allowing us to assess the benefit of the structural pathway at comparable data exposure. They do not establish an optimal allocation of a fixed FLOP budget, and we will clarify this distinction. Among the configurations evaluated, OptimRNA-34M provides a particularly favorable training-cost–performance trade-off. Increasing model size to 88M yields smaller structural-prediction gains than introducing the pair pathway into the 34M backbone. These observations support the smaller structure-aware model as a promising configuration under limited resources, although they do not determine whether a token-only backbone trained longer could close the gap.

Our additional experiments requested by Reviewer j4fN compare OptimRNA-88M with ERNIE-RNA retrained using the same recipe and approximately 202k updates. Under the validation-selected single-attention-map readout, OptimRNA achieves AUPRCs of 0.565 versus 0.552 on TS0 and 0.545 versus 0.486 on bpRNA-new, with comparable performance on Rivas TestSetB and higher performance on RNA3DB. The published ERNIE-RNA checkpoint reaches 0.585 and 0.614, respectively, but its reported pretraining required 20 days on 24 V100 GPUs, compared with approximately 40 hours on one H100 for OptimRNA-88M and 90 hours for our retrained ERNIE-RNA. These hardware-dependent figures are not a FLOP-normalized comparison, but illustrate the substantial difference in training resources. The published checkpoint’s stronger performance motivates investigating longer pretraining, although differences between recipes prevent attributing this gap to training duration alone. We are therefore extending OptimRNA-88M pretraining from approximately 202k to 606k gradient steps and will report the results when available. This experiment will assess whether the larger model benefits from additional training exposure. A comprehensive comparison of how best to allocate equal compute across architecture, model size, and training duration remains beyond the present evaluation; we will accordingly limit our efficiency claims to the parameter counts and measured computational costs supported by our experiments.

Q3

We include the baselines and the ablation on the antidiagonal convolution in our reply to Q1, with the statistical significance of the difference in performance compared with the full OptimRNA model. Removing the antidiagonal convolution significantly hurts performance across all datasets in the zero-shot setting, as does removing the pair state entirely. We will add statistical significance in the ablations in the paper.

AuthorsOpenReview ↗

Answer to Reviewer Lnso [part 2]

Q4

Our paper investigates whether an RNA-specific pair-state architecture improves representation learning under sequence-only masked language modeling. Accordingly, our zero-shot performance claims are explicitly restricted to the evaluated RNA language models; we do not claim state-of-the-art performance across all structure-prediction methods. AlphaFold3 learns from three-dimensional structural supervision. Comparing secondary structures extracted from its predictions would therefore address a broader question about predictive performance across different supervision regimes, rather than our central question about structural information learned through sequence-only pretraining. We consider this additional benchmark outside the scope of the present paper. For supervised adaptation, the manuscript already acknowledges that differences in prediction heads and optimization sensitivity prevent interpreting published scores as each model’s optimal performance. Our conclusions therefore emphasize the performance–parameter trade-off and controlled comparisons between OptimRNA variants under shared training and evaluation procedures. Establishing a broader ranking would require extensive, computationally costly tuning across architectures and model sizes. We believe the field would benefit from systematically investigating parameter-efficient fine-tuning strategies that transfer reliably across scales, making future comparisons more accessible and reproducible; developing and validating such protocols falls outside the scope of this work.

The response addresses the scope of the zero-shot claims, not my question. I asked for the supervised fine-tuned model to be benchmarked against AlphaFold3 as a reference point; running AF3 on the same held-out split and scoring its predicted structures with the same DSSP-based script needs no tuning on the AF3 side, so cost is not a reason to omit it. We are not asking for a claim of superiority over structure-supervised methods, only for a number readers can calibrate against.

Reviewer_j4fNOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes OptimRNA, a structure-aware RNA language model that maintains a persistent pairwise representation alongside conventional token representations during masked language model (MLM) pretraining. The pair state is initialized using RNA base-pairing constraints and progressively updated through shared signed low-rank transformations and antidiagonal convolutions. The accumulated representation is subsequently incorporated into the attention mechanism as a bounded bias. The authors also develop a factorized implementation to reduce the memory overhead of pairwise modeling.

The method achieves promising performance on RNA secondary-structure prediction benchmarks, including zero-shot prediction on bpRNA TS0 and leave-one-family-out evaluation on ArchiveII. The paper further presents architectural ablations, layerwise mechanistic analyses, evaluations on downstream tasks less directly related to secondary structure, and computational efficiency measurements.

Critical strengths

1. Biologically motivated and computationally efficient pairwise modeling.

The proposed antidiagonal convolution is well motivated by the geometry of RNA stems, where consecutive base pairs follow antidiagonal patterns in a pairwise contact map. The shared signed low-rank parameterization also provides a lightweight alternative to dense multichannel pair representations. I appreciate the effort to incorporate RNA-specific structural inductive biases into sequence-only pretraining while keeping the architecture relatively efficient.

2. Promising empirical results and useful matched ablations.

The paper provides encouraging results on secondary-structure prediction, particularly in the low-parameter regime. On bpRNA TS0, the matched 34M backbone improves from 0.5049 PR-AUC without the pair pathway to 0.5315 with the pair pathway, and further to 0.5659 with antidiagonal propagation. The ArchiveII leave-one-family-out results are also promising. For OptimRNA-34M, the average F1 increases from 66.9% for the matched baseline to 72.8% with the pair pathway, and further to 77.3% when the pair state is directly provided to the downstream prediction head. These results support the usefulness of the proposed structural pathway under the evaluated settings.

3. Useful mechanistic analysis.

I appreciate the analysis of how structural information evolves across layers. In particular, Figure 4(a) shows that the pair state's direct structure-prediction performance improves substantially in later layers, providing evidence that the learned representation contains useful structural information. The additional analyses of effective rank and the relationship between pair representations and token embeddings also help characterize the proposed mechanism.

Critical weaknesses

W1. Limited conceptual novelty relative to existing pairwise representation architectures, particularly ERNIE-RNA.

The central idea of propagating pairwise structural information across network layers has substantial precedent. AlphaFold2's Evoformer and AlphaFold3's Pairformer already employ persistent pair representations and pair-informed attention. More directly, ERNIE-RNA incorporates biologically motivated pairing biases into MLM pretraining without structural supervision and passes pair-indexed attention information between consecutive Transformer layers.

I acknowledge that OptimRNA introduces meaningful architectural differences, including a dedicated scalar pair state, shared signed low-rank updates, learned decay, and RNA-specific antidiagonal propagation. In particular, its explicitly accumulated pair state is distinct from ERNIE-RNA's propagation of attention maps. Nevertheless, the conceptual contribution appears closer to an efficient, RNA-specific refinement of existing pairwise representation mechanisms than a fundamentally new representation-learning paradigm.

Given this relationship to prior work, it is important to establish the specific advantages of the proposed architecture over closely related alternatives. However, the comparison with ERNIE-RNA does not isolate the architectural contribution, since the models differ in backbone design, pretraining data, optimization, and implementation. Although OptimRNA achieves better zero-shot results under the authors' unified nucleotide-dependency probe, the improvement over ERNIE-RNA is relatively modest, particularly in F1, and its statistical robustness is not established. A more controlled comparison with ERNIE-style attention propagation under a matched backbone and training recipe would substantially strengthen the evidence for the proposed architectural contribution.

W2. Insufficient evidence that a learned, persistent pair state is necessary over simpler structural-prior injection.

The pair state is initialized using a handcrafted structural prior that identifies candidate stems containing at least four consecutive Watson–Crick complementary pairs: Pij0=αΠij.P^0_{ij} = \alpha \Pi_{ij}. This initialization incorporates biologically meaningful pairing information before learning. The state is subsequently refined through learned low-rank updates, antidiagonal propagation, and cross-layer accumulation.

The key question is whether the improvements specifically arise from learning and persistently accumulating pairwise information, rather than from introducing structural priors into attention or learning independent pairwise biases at individual layers.

The existing ablations compare a token-only baseline, a pair-state model without antidiagonal propagation, and the complete model. Although these experiments are useful, they do not sufficiently isolate the contributions of the initial structural prior, learned pairwise updates, and persistent cross-layer accumulation. In particular, three controls are missing:

  1. Fixed-prior baseline: A matched Transformer that uses the initial structural prior P0P^0 as an attention bias without learned pairwise updates.
  2. Non-persistent pairwise baseline: A model that computes learned pairwise attention biases independently at each layer, without maintaining an accumulated pair state.
  3. Zero-initialization baseline: A model retaining the learnable persistent pair pathway but initializing P0=0P^0=0.

Figure 4(a) provides evidence that the learned pair state becomes more structurally informative than its initialization, particularly in later layers. However, this does not establish that persistent accumulation is necessary or superior to simpler alternatives.

Furthermore, the substantial improvement obtained by supplying the pair state directly to the prediction head on ArchiveII raises a related question: how much of this gain requires a learned pair representation, rather than simply supplying the handcrafted structural prior as an additional input channel?

Since persistent pair-state refinement is a central contribution of the paper, I consider these missing comparisons a significant limitation.

W3. Insufficient specification of pair-prior initialization during MLM pretraining and potential information leakage.

The initial pair state is constructed from sequence-dependent nucleotide complementarity constraints, while MLM pretraining requires reconstructing corrupted nucleotide identities. However, the manuscript does not clearly specify whether P0P^0 is constructed from the original sequence before masking or from the corrupted sequence after masking.

This distinction is important. If P0P^0 is computed from the original sequence before MLM corruption, it may reveal information about the nucleotide identities that the model is supposed to predict, creating an unintended information pathway through pair-biased attention. The concern is particularly subtle because the initialization depends on runs of at least four consecutive complementary pairs, rather than only on the nucleotide identities at the two endpoints of each pair-state entry. Consequently, even removing pair-state rows and columns associated with masked positions might not eliminate indirect information leakage through neighboring candidate-stem entries.

If P0P^0 is constructed entirely from the corrupted input sequence, this particular leakage concern can be avoided, provided the original nucleotide identities are not used elsewhere in constructing the prior. However, the resulting structural prior would be incomplete around masked positions, which may influence the model's behavior during pretraining.

I am not claiming that information leakage occurs. Rather, the current methodological description is insufficient to rule it out. Given that the model's structural prior is sequence-dependent and directly influences the MLM computation, the authors should explicitly clarify the order of masking and prior construction, including the handling of masked and randomly replaced positions. Pseudocode or a corresponding implementation reference would be helpful.

W4. Incomplete empirical comparison with ERNIE-RNA and insufficient evaluation of generalization.

(a) Zero-shot structure prediction.

The paper evaluates RNA language models using a unified nucleotide-dependency probe, which is useful for comparing representations under a common evaluation procedure. Under this protocol, OptimRNA-88M achieves better results than ERNIE-RNA on bpRNA TS0, with PR-AUC values of 0.5714 versus 0.5566 and F1 scores of 52.5% versus 51.7%.

However, ERNIE-RNA's original work also demonstrates a native zero-shot structure-prediction approach that directly interprets its learned attention maps. The original ERNIE-RNA paper reports a binary zero-shot F1 of 0.549 on bpRNA-1m using this approach. This result is not directly comparable with OptimRNA's zero-shot F1 because the evaluation metrics, prediction procedures, and decoding protocols differ. Nevertheless, it highlights an important distinction: the current experiments demonstrate an advantage for OptimRNA under the selected common probe, but do not establish an advantage over ERNIE-RNA's native zero-shot structural readout.

A more complete comparison would evaluate the native pairwise representations of both architectures under harmonized metrics and decoding procedures, while explicitly accounting for any head-selection or threshold-selection steps.

(b) Fine-tuned structure prediction.

Despite the proposed architectural improvements, OptimRNA does not outperform ERNIE-RNA on the main fine-tuned bpRNA TS0 benchmark. As shown in Table 1, ERNIE-RNA achieves a shift-tolerant F1 of 79.1% when its released fine-tuned checkpoint is re-evaluated under the authors' protocol, whereas OptimRNA-88M achieves 77.6%, including when the pair state is directly provided to the prediction head.

I recognize that these results are not a fully controlled architectural comparison, since ERNIE-RNA uses its own attention-map prediction head and the fine-tuning procedures differ. Nevertheless, the results do not demonstrate a consistent downstream performance advantage for OptimRNA over its closest architectural competitor.

Additionally, Appendix D.1 states that the authors' bpRNA fine-tuning results are obtained from a single seed. Reporting variability across fine-tuning runs, or confidence intervals for key performance differences, would help assess the robustness of the comparisons.

(c) Missing evaluation on challenging out-of-distribution benchmarks.

ERNIE-RNA evaluates secondary-structure prediction on several challenging benchmarks beyond bpRNA TS0, including RIVAS TestSetB, bpRNA-new, and RNA3DB-2D. These datasets provide complementary tests of generalization. RIVAS TestSetB evaluates transfer under different RNA family compositions, bpRNA-new tests generalization to RNA families absent from the supervised training benchmark, and RNA3DB-2D uses structurally dissimilar data splits to assess generalization beyond familiar structural patterns.

OptimRNA primarily evaluates generalization through ArchiveII leave-one-family-out experiments. While this is a valuable evaluation of cross-family transfer, it does not fully characterize performance on independent datasets with different family compositions or structural distributions.

Given that the proposed pair-state mechanism is intended to improve transferable structural representations, evaluation on these established benchmarks would considerably strengthen the paper. Without such results, it remains unclear whether the reported advantages persist under more challenging distribution shifts, particularly on datasets where ERNIE-RNA has already demonstrated strong performance.

Questions for authors

Q1. Structural prior, learned pairwise updates, and persistent accumulation. Could the authors provide controlled comparisons with the following variants?

  1. A model that uses the fixed structural prior P0P^0 as an attention bias without learned pairwise updates.
  2. A model that computes learned pairwise biases independently at each layer without cross-layer accumulation.
  3. A model that retains the learnable persistent pair pathway but initializes P0=0P^0=0.

For the ArchiveII experiments, it would also be informative to compare the learned P→headP \rightarrow \mathrm{head} variant against a prediction head receiving the fixed structural prior P0P^0. These controls would help identify which aspects of the pair pathway are responsible for the observed improvements and, in particular, whether persistent accumulation provides benefits beyond simpler structural-prior injection.

Q2. MLM masking and pair-prior initialization. Is the initial pair state constructed before or after MLM corruption? Could the authors provide pseudocode describing how masked, randomly replaced, and unchanged selected positions are handled when constructing candidate stems? In particular, if the initial prior uses information from the original sequence, how do the authors prevent masked nucleotide identities from being indirectly exposed through neighboring candidate-stem entries?

Q3. Controlled comparison with ERNIE-RNA and fine-tuning protocols. Could the authors clarify how OptimRNA would compare with ERNIE-style residual attention propagation under an otherwise matched backbone and pretraining setup? Such a comparison would help isolate the contribution of the dedicated persistent pair state from differences in backbone design and training recipe. Additionally, OptimRNA reports RiNALMo's published fine-tuned F1 of 74.7% on bpRNA TS0, whereas AIDO.RNA reports 77.2% after independently fine-tuning RiNALMo under its downstream training procedure. Although the manuscript discusses differences in fine-tuning settings, could the authors clarify the implications of these alternative reported baseline results for the relative performance comparisons? In particular, it would be helpful to distinguish improvements attributable to pretraining representations from those potentially influenced by downstream adaptation protocols.

Q4. Generalization performance and robustness. Could the authors evaluate OptimRNA on RIVAS TestSetB, bpRNA-new, and RNA3DB-2D using the corresponding established training splits and comparable evaluation protocols? I am particularly interested in whether the advantages of the learned pair representation persist on unseen RNA families and structurally dissimilar test sets. It would also be useful to report uncertainty estimates for the key comparisons, particularly where performance differences between OptimRNA and closely related baselines are relatively small.

Q5. Pretraining sampling, effective data coverage, and sequence homology. Appendix B describes a training corpus of approximately 38.8 million sequences and a pretraining schedule of 202k steps with a batch size of 192, described as a single pass over the corpus. However, the authors also employ family-damped sampling, which substantially changes the proportions of RNA types encountered during training. Could the authors clarify how the single-pass description relates to this non-uniform sampling procedure? Specifically:

  1. Are sequences sampled with replacement, and what fraction of the unique training sequences is actually encountered during pretraining?
  2. How frequently are sequences from underrepresented RNA types repeated, and are some sequences from overrepresented RNA types never sampled?
6 comments
AuthorsOpenReview ↗

Answer to Reviewer j4fN

Thank you for your thorough and constructive assessment of our work. Below, we address each question and its associated weaknesses. In response to Q4, we have extended the evaluation to additional generalization benchmarks (bpRNA-new, Rivas TestSetB and RNA3DB-2D) and incorporated statistical significance tests into the comparisons reported below. We will also include these analyses in the revised manuscript.

To contextualize generalization across these benchmarks, we additionally quantified differences in RNA-family composition relative to the pretraining dataset. We assigned each sequence in the benchmark datasets the RNACentral type label of its most similar sequences using mmseqs2 with a similarity cutoff of 90%. We then computed the total variation distance between benchmark and pretraining type-frequency distributions, considering both the original corpus and the distribution actually encountered during training using family sampling temperature of 0.7.

BenchmarkTV distance to sampled pretraining distribution (Sampling temperature = 0.7)TV distance to original corpus (no special sampling)
bpRNA TS00.690.83
RNA3DB-2D test0.760.84
Rivas TestSetB0.870.93
bpRNA-new0.930.97

Lower distances indicate more similar family composition. All four benchmarks differ substantially from pretraining, with bpRNA-new showing the largest shift under this measure. Our sampling scheme reduces this discrepancy for every benchmark. These estimates characterize family composition rather than sequence-level homology or structural novelty, and inferred family labels are proxies.

Q1 (W2)

Thank you for suggesting these controls. We trained all three requested variants using the same 34M backbone, pretraining data, and training recipe, and evaluated them on four benchmarks using the paper’s nucleotide-dependency readout. The table reports mean per-sequence AUPRC (%). Parentheses indicate differences from full OptimRNA in percentage points. Asterisks denote Holm-adjusted significance from two-sided paired Wilcoxon signed-rank tests against full OptimRNA (* (p<0.05), ** (p<0.01), *** (p<0.001)). We drop sequences from the datasets that have no base pairs annotated.

ModelTS0 (n=1304)bpRNA-new (n=5401)Rivas TestSetB (n=430)RNA3DB-2D (n=147)
Full OptimRNA56.5945.8172.8258.37
OptimRNA with zero-prior (item 3)56.13 (−0.46)47.21 (+1.41)***69.48 (−3.34)**52.03 (−6.35)**
OptimRNA without cross-layer accumulation (item 2)55.63 (−0.96)*46.11 (+0.30)70.60 (−2.22)*55.27 (−3.11)
OptimRNA without antidiagonal conv53.16 (−3.43)***40.90 (−4.91)***71.10 (−1.71)***53.11 (−5.26)***
Baseline, with attention bias (item 1)50.12 (−6.46)***34.05 (−11.75)***67.92 (−4.90)***50.04 (−8.33)***
Baseline, no attention bias50.50 (−6.09)***35.78 (−10.02)***65.50 (−7.31)***49.12 (−9.25)***

Removing the initial prior (item 3) slightly decreases performance on TS0, significantly reduces it on Rivas TestSetB and RNA3DB, and significantly improves it on bpRNA-new. The latter exhibits the largest RNA-family composition shift from pretraining, suggesting that the prior may favor pretraining-distribution regularities at some cost to transfer. This motivates investigating structural priors jointly with data curation and sampling in future work.

Removing cross-layer accumulation (item 2) discards earlier updates while retaining each update’s influence on the next layer, omitting λℓPℓ−1\lambda_\ell P^{\ell-1} from Eq. 7. Performance decreases significantly on TS0 and Rivas TestSetB, with no significant difference on the other benchmarks. Removing antidiagonal convolution significantly reduces performance on all four datasets, demonstrating a more consistent benefit from stem-aware propagation.

Using only a fixed-prior attention bias (item 1) yields mixed changes relative to the token-only backbone. Full OptimRNA significantly outperforms both baselines on every benchmark, showing that prior injection alone cannot explain its gains.

For ArchiveII, we repeated OptimRNA-34M’s leave-one-family-out fine-tuning protocol, supplying P0P^0 alongside token embeddings to the prediction head instead of the final learned PP. Across three seeds, this yields 73.0±1.073.0\pm1.0 mean F1 score, compared with 72.8±0.572.8\pm0.5 for token embeddings alone and 77.3±0.377.3\pm0.3 with learned PP. The fixed prior adds little benefit, whereas the learned state substantially improves cross-family prediction.

Together, these controls support learned pairwise refinement and stem-aware propagation as OptimRNA’s central strengths, with additional, dataset-dependent benefits from initialization and accumulation. The complete architecture achieves the highest mean AUPRC on three of the four zero-shot benchmarks.

AuthorsOpenReview ↗

Answer to Reviewer j4fN [part 2]

Q2 (W3)

Thank you for highlighting this important detail. The pair state is constructed after MLM corruption, so information about the original sequence is not leaked through P0P^0. When the masked token is randomly replaced or left unchanged from the original nucleotide, P0P^0 is calculated using the randomly replaced nucleotide or the original nucleotide. We will clarify these aspects in the paper.

Here is the pseudocode showing how P0P^0 is computed:

    # x contains the token ids of the corrupted sequence
    # k is the minimum stem length (4 by default)
    # s is the seed strength (5 by default)
    
    Function P0(x[1..L]):			    
        # Recover nucleotides, if masked or special token, assign none
        n[i] = {A,C,G,U}[x[i]] if x[i] ∈ {A,C,G,U}, else none

        # Canonical pair-mask
        C[i,j] = (n[i],n[j]) ∈ {AU, UA, CG, GC}

        # Only preserve upper triangle
        U[i,j] = C[i,j] and i < j

        # Find stem starts
        S[i,j] = AND_{t=0..k-1}  U[i+t, j-t]

        # Mark every cell in a stem
        R[i,j] = OR_{t=0..k-1}   S[i-t, j+t]

        Return s * (R OR R^T)

Q3 (W1, W4) (part 1)

Controlled pretraining comparison. We have trained an ERNIE-RNA backbone using the same training recipe as OptimRNA. To match its size, we compare it with our 88M checkpoint. The ERNIE-RNA retraining took about 90 GPU hours on 1 H100, compared with 40 GPU hours for OptimRNA-88M. We also compare this model to the published version of ERNIE-RNA, trained for much longer than these models, namely 20 days with 24 V100 GPUs as reported by the authors.

Common nucleotide-dependency readout. We selected nucleotide dependency as our default because all evaluated RNA-LMs share the masked-nucleotide prediction objective. It extracts pairwise dependencies from changes in these predictions under nucleotide perturbations, providing a common interface without fitting a downstream predictor or selecting attention heads using structural labels. Attention-based readouts, by contrast, depend more directly on architectural choices, particularly when attention explicitly incorporates pairing information, as in ERNIE-RNA. Our default therefore compares structural information recoverable through a shared prediction task, without assuming that it is each model’s optimal structural readout. All results below are AUPRC (%), with differences from OptimRNA-88M in parentheses and the Holm-adjusted (within each table) statistical significance from two-sided paired Wilcoxon signed-rank tests is reported (* p < 0.05, ** p < 0.01, *** p < 0.001).

ModelTS0bpRNA-newRivas TestSetBRNA3DB-2D test
OptimRNA-88M57.151.670.655.4
ERNIE-RNA (retrained)55.0 (-2.1)***44.0 (-7.6)***70.5 (-0.1)50.2 (-5.2)**
ERNIE-RNA (published)55.7 (-1.5)***59.7 (+8.1)***71.3 (+0.8)53.3 (-2.1)

Under this common readout, OptimRNA exceeds retrained ERNIE-RNA on all four benchmarks, with a small difference on Rivas TestSetB. Compared with published ERNIE-RNA, OptimRNA performs better on TS0 and RNA3DB-2D, while ERNIE-RNA performs better on bpRNA-new and Rivas TestSetB.

AuthorsOpenReview ↗

Answer to Reviewer j4fN [part 3]

Q3 (W1, W4) (part 2)

Model-specific attention-based structural readout. To complement this comparison, we evaluated ERNIE-RNA’s zero-shot approach by using an attention map and applying the E2Efold post-processing pipeline. We select the best attention map using bpRNA-VL0 consistently across models. This assesses structural information directly available in attention, although head selection requires structural validation labels.

ModelTS0bpRNA-newRivas TestSetBRNA3DB-2D test
OptimRNA-88M56.554.570.256.0
ERNIE-RNA (retrained)55.2 (-1.3)48.6 (-6.0)***70.4 (+0.3)53.0 (-2.9)
ERNIE-RNA (published)58.5 (+1.9)***61.4 (+6.9)***71.6 (+1.5)***55.3 (-0.6)

For OptimRNA, the shared pair-state bias can inform multiple attention heads, motivating a complementary readout that averages several maps. We therefore extended the protocol to average the top-kk attention maps before post-processing, selecting both the maps and kk on bpRNA-VL0. We applied the same selection procedure to all three models.

ModelkTS0bpRNA-newRivas TestSetBRNA3DB-2D test
OptimRNA-88M358.357.470.660.3
ERNIE-RNA (retrained)155.2 (-3.1)***48.6 (-8.9)***70.4 (-0.2)53.0 (-7.3)***
ERNIE-RNA (published)258.9 (+0.6)61.3 (+3.9)***72.0 (+1.4)*54.0 (-6.3)***

Combining heads improves OptimRNA across all four benchmarks, whereas its effects on published ERNIE-RNA are mixed. OptimRNA then exceeds retrained ERNIE-RNA throughout with a significant advantage for three out of four datasets, approaches the published checkpoint on TS0, and retains a clear advantage on RNA3DB-2D; published ERNIE-RNA remains stronger on bpRNA-new and Rivas TestSetB. These complementary evaluations distinguish performance under a common MLM-based readout from structural information recoverable through validation-selected attention maps.

Fine-tuning. We also fine-tuned these three models on bpRNA TR0 and evaluated them on TS0 and bpRNA-new. For the sake of efficiency, we freeze the backbones, and only train a RiNALMo-style structure prediction head on the last-layer embeddings for 20 epochs. The best epoch is selected using the bpRNA validation set. We report average AUPRC (%) over three seeds.

ModelTS0bpRNA-new
OptimRNA-88M68.857.9
ERNIE-RNA (retrained)67.3 (-1.4)***52.4 (-5.5)***
ERNIE-RNA (published)70.8 (+2.0)***62.7 (+4.8)***

We see the same trend as in the zero-shot experiment: for the same training budget, OptimRNA outperforms ERNIE-RNA, but ERNIE-RNA’s published version pulls ahead. Overall, we suspect that with more training, OptimRNA could close this gap. Consequently, we are currently training OptimRNA with a larger training budget, and we will share the results as soon as possible.

Implications of fine-tuning protocols. Thank you for highlighting RiNALMo’s alternative 77.2% F1 result. We will report it alongside the original 74.7%, explicitly distinguishing the adaptation protocols. Against this stronger reference, OptimRNA-88M’s 77.6% demonstrates competitive performance at a substantially smaller model size; the 0.4-point difference does not establish superior pretrained representations. Our setup combines a RiNALMo-style head and progressive unfreezing with AIDO.RNA’s 60-epoch budget. Preliminary runs underfit when newly unfrozen layers received one tenth of the already-decayed learning rate. We therefore use one tenth of the base learning rate, with base rates of 5×10−45\times10^{-4} for 34M and 2.5×10−42.5\times10^{-4} for 88M. OptimRNA-88M reaches 77.6% with either the token-only or token-plus-pair-state head. This sensitivity motivates our emphasis on a common zero-shot readout and matched architectural ablations. Supervised scores describe the evaluated pipelines, rather than each model’s optimal adaptation. Establishing a broader ranking would require computationally costly tuning across architectures and scales. Systematic investigation of parameter-efficient fine-tuning could make such comparisons more accessible, but developing those protocols lies outside this work’s scope.

Reviewer_DbJ6OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces a persistent pair-state pathway that explicitly models nucleotide-pair relationships alongside standard token representations, allowing structural information to accumulate and be refined across transformer layers. This work proposes OptimaRNA -- an efficient, RNA-specific mechanism for updating the pair state using signed low-rank pairwise writes and antidiagonal propagation, which captures the characteristic geometry of stacked base pairs in RNA stems.

The comprehensive evaluation and ablation studies demonstrate that the resulting structure-aware RNA language model improves zero-shot secondary-structure prediction and cross-family structural generalization, while maintaining competitive performance on broader RNA tasks and substantially lower memory usage than dense pair-representation approaches.

Critical strengths

I find the empirical validation particularly convincing. The ablation studies systematically demonstrate the effectiveness of the major design choices, with each component yielding measurable improvements. Moreover, the gains extend to zero-shot secondary-structure prediction and, importantly, generalization across unseen RNA families. This result provides strong evidence that the proposed architecture captures meaningful and transferable structural information, which I view as a key strength of the paper.

Critical weaknesses

A key weakness is that the discussion and positioning of existing structure-aware RNA language models could be more precise. The paper emphasizes that OptimRNA incorporates biological pairing knowledge while retaining sequence-only MLM as the training objective, but this property is not unique to the proposed approach. For example, some prior methods cited by the authors, such as Wang et al. (2024), also incorporate biological knowledge into pretraining while retaining a sequence-only MLM objective. Therefore, the related-work discussion does not sufficiently clarify what fundamentally distinguishes the proposed method from these existing approaches. I believe the paper would benefit from a more systematic comparison of where structural knowledge is introduced in each method, whether structural annotations are required during training, and what architectural or representational advantage the proposed persistent pair-state mechanism provides over prior structure-aware pretraining strategies.

But I believe the proposed method in this work still would outperform those works.

Questions for authors

Authors mentioned multiple RNA languange models pre-trained using structure information or priors, however they only compared their methods with RiNALMo, SpliceBERT, ERNIE-RNA, and some task-specific algorithms such as LinearFold. I think there might need some comprehensive comparison with other pre-trained models, epsecially prior information leveraged for pre-training and whether this prior information would be needed for prediction.

In some of the experiments, we can see larger models could outperform the proposed method on some metric. I am particularly interested in whether the authors have ever tried to scale-up their recipe in terms of model size (except only two checkpoints of 34M and 88M).

1 comment
AuthorsOpenReview ↗

Answer to Reviewer DbJ6

We thank you for your positive assessment and your suggestions to improve our work.

Weaknesses. Thank you for the feedback and for suggesting a clearer comparison with prior work. We agree that incorporating biological knowledge while retaining MLM is not unique to OptimRNA. RNAErnie uses motifs to guide masking, structRFM uses secondary structures to guide masking, and ERNIE-RNA carries pairwise attention scores across layers and uses a biologically informed prior attention bias. OptimRNA’s contribution is a separate, symmetric pair state that accumulates learned low-rank updates along RNA stem patterns and is available as a direct downstream readout. We will sharpen this distinction in the main text and add a supplementary comparison of where priors enter each method, whether pretraining uses structural annotations or predictions, and what inputs are needed at inference.

Q1.

  • Zero-shot structure prediction. On bpRNA TS0, we compare with 10 RNA-LM checkpoints from six model families, counting the published and retrained RiNALMo-micro checkpoints separately, as well as four standalone folding methods (See Table 8 in the appendix). The RNA-LM comparison already includes RNAErnie, structRFM, and ERNIE-RNA, which cover motif-aware masking, structure-guided masking, and architectural pairing biases, respectively. None of these models require additional information at prediction time; instead, they use biological priors to guide training or inform architectural changes. We apply the same nucleotide-dependency readout to all RNA-LMs to compare them without fitting a downstream predictor. For OptimRNA, we compute the pairing prior from the input sequence at inference; no external structure annotation or folding prediction is required. In the main text, we report only a subset of models for conciseness; we chose AIDO.RNA and RiNALMo-giga as they are the largest available models, ERNIE-RNA as it has the most comparable architecture, and RNA-FM, which uses no specific biological prior, as a similar-sized baseline to ERNIE-RNA and OptimRNA-88M.

  • Transfer beyond structure prediction. Splice-site and nucleotide-modification prediction test whether the pair pathway compromises performance on tasks less directly tied to secondary structure. The primary comparison is with matched OptimRNA backbones without the pair state. RiNALMo provides a general-purpose reference, including a checkpoint retrained with our recipe; ERNIE-RNA provides the closest architectural reference; and SpliceBERT and Spliceator provide task-specific references for splicing. We will clarify that these experiments test for adverse transfer on two task groups.

Q2.

We have not tested model sizes larger than the 88M-parameter checkpoint because we designed OptimRNA with efficiency in mind and observed diminishing returns in model scale in the existing literature. For example, AIDO.RNA’s 650M checkpoint achieves an AUPRC of 0.5273 on the zero-shot TS0 task, while its 1.6B checkpoint reaches 0.5281. As far as we can tell, the AIDO.RNA authors trained these two models for the same number of gradient steps. This is similar to what we observe with OptimRNA-34M and 88M: the gains coming from scaling model size under the same gradient step budget seem to be marginal. However, our retrained version of ERNIE-RNA requested by Reviewer j4fN, also trained for ~202k steps, and its published version, trained for considerably longer, show a wider performance gap, especially on bpRNA-new. Our retrained version achieves 0.550 and 0.440 AUPRC on bpRNA TS0 and bpRNA-new respectively, whereas the published version achieves 0.557 and 0.597. If we increase the compute budget overall, it seems that training longer is a more promising path than training larger models, or at least both training time and model size need to increase together. We are currently training OptimRNA-88M for 606k steps to test this hypothesis, and we will report back on its performance at this larger training budget.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Shared reply on longer-trained OptimRNA-88M model

Dear Reviewers,

Thank you for your comments on model scaling, training efficiency, and comparisons with ERNIE-RNA. We have now extended OptimRNA-88M pretraining from approximately 200k to 600k updates, keeping its architecture and parameter count unchanged. This experiment assesses whether the larger model benefits from additional training exposure (Reviewer DbJ6, Q2), strengthens the comparison with the longer-trained published ERNIE-RNA checkpoint (Reviewer j4fN, Q3), and complements our analysis of training efficiency (Reviewer Lnso, Q2).

Following our response to Reviewer j4fN (Q3), we evaluate secondary-structure prediction on four benchmarks using three complementary readouts, without training a downstream predictor:

  • Nucleotide dependency (ISM) provides a common evaluation interface through the MLM prediction task, without selecting architecture-specific attention maps. Default used in the paper.
  • Top-1 attention, following ERNIE-RNA’s approach, directly exploits attention maps shaped by its structural inductive bias.
  • Top-3 attention averages top-k maps (1≤k≤31 \leq k \leq 3), motivated by OptimRNA’s accumulated pair state informing multiple attention heads across later layers.

We apply all three protocols to every model. Attention maps are selected on bpRNA-VL0 using structural labels; ISM requires no such selection. We report mean per-sequence AUPRC (%), highlighting the highest score across all models and readouts separately for each benchmark.

ModelReadoutbpRNA TS0bpRNA-newRivas TestSetBRNA3DB-2D
OptimRNA-88M, 200k updatesISM57.151.670.655.4
Top-154.749.569.354.6
Top-357.153.469.958.4
OptimRNA-88M, 600k updatesISM59.559.373.154.5
Top-158.559.772.057.0
Top-359.959.772.255.7
ERNIE-RNA, retrained 200kISM55.044.070.550.2
Top-155.747.272.054.8
Top-353.843.271.450.4
ERNIE-RNA, publishedISM55.759.771.353.3
Top-159.561.973.558.8
Top-359.861.374.257.4

Matched pretraining demonstrates OptimRNA’s efficiency. At approximately 200k updates under the same pretraining recipe, OptimRNA-88M exceeds retrained ERNIE-RNA on all four benchmarks under the common ISM readout, while requiring 40 versus 90 hours on one H100—a 2.25× training-speed advantage. This establishes stronger structural prediction at matched training exposure and lower computational cost. The extended OptimRNA run required approximately 110–120 H100 hours, enabling threefold training exposure for only 1.22–1.33× the training time of retrained ERNIE-RNA. Although these are not exactly equal-time runs, they illustrate the additional training exposure enabled by OptimRNA’s faster implementation.

Longer training strengthens structural prediction and transfer. Extending OptimRNA pretraining improves performance in 10 of the 12 dataset–readout combinations, including every readout on TS0, bpRNA-new, and Rivas TestSetB. The largest gains occur on bpRNA-new, the benchmark with the greatest RNA-type composition shift from pretraining, including a 7.7-point AUPRC increase under ISM. The extended checkpoint also matches or exceeds retrained ERNIE-RNA across all twelve combinations. These results support allocating the architecture’s computational savings to additional training exposure.

The extended checkpoint is increasingly competitive with published ERNIE-RNA. Under ISM, OptimRNA now surpasses that checkpoint on three benchmarks and nearly matches it on bpRNA-new. Across all readouts, it achieves the highest observed TS0 score, while published ERNIE-RNA remains strongest on the other three datasets. The latter required 20 days on 24 V100 GPUs, substantially more training resources than our runs, although different hardware prevents a direct compute-equivalent comparison. These findings show that OptimRNA’s initial checkpoint left considerable gains available through longer pretraining. The mixed changes on RNA3DB-2D nevertheless motivate jointly improving training exposure, data curation, and sampling to make these gains more consistent across distributions.

Together, these experiments support OptimRNA as an efficient architecture that converts additional training into increasingly competitive structural predictions. Its measured training-speed advantage and memory-efficient pair formulation provide a practical basis for further scaling training exposure and sequence length, with longer RNAs and viral genomes as promising future applications. We will incorporate these results and the distinctions between readout protocols into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.