OptimRNA: Structure-Aware RNA Language Modeling with Factorized Pair States

RNA function is strongly shaped by pairwise structure, yet RNA language models (RNA-LMs) rely on BERT-style Transformers that only maintain per-token representations between layers. To explicitly integrate prior knowledge of RNA pairing into sequence learning, we introduce OptimRNA, an architecture that augments the token stream with a pair state maintained and refined across Transformer layers. Initialized from canonical pairs supported by stem-compatible neighborhoods, the pair state is progressively updated from contextual token representations using signed low-rank transformations propagated by antidiagonal convolutions encoding RNA-stem patterns. Every Transformer layer updates the pair state, while later layers read it as a bounded attention bias. Its low-rank formulation enables memory-efficient pair modeling with negligible parameter overhead compared with standard RNA-LMs. We show that OptimRNA outperforms all evaluated RNA-LMs in zero-shot secondary-structure prediction and, after supervised fine-tuning, competes with substantially larger foundation models without compromising performance on tasks less directly tied to secondary structure. Controlled ablations and mechanistic analyses support the complementary roles of the pair pathway and stem-aware propagation. Together, these results establish OptimRNA as an efficient biologically-informed architecture for RNA language modeling.