W1. Limited conceptual novelty relative to existing pairwise representation architectures, particularly ERNIE-RNA.
The central idea of propagating pairwise structural information across network layers has substantial precedent. AlphaFold2's Evoformer and AlphaFold3's Pairformer already employ persistent pair representations and pair-informed attention. More directly, ERNIE-RNA incorporates biologically motivated pairing biases into MLM pretraining without structural supervision and passes pair-indexed attention information between consecutive Transformer layers.
I acknowledge that OptimRNA introduces meaningful architectural differences, including a dedicated scalar pair state, shared signed low-rank updates, learned decay, and RNA-specific antidiagonal propagation. In particular, its explicitly accumulated pair state is distinct from ERNIE-RNA's propagation of attention maps. Nevertheless, the conceptual contribution appears closer to an efficient, RNA-specific refinement of existing pairwise representation mechanisms than a fundamentally new representation-learning paradigm.
Given this relationship to prior work, it is important to establish the specific advantages of the proposed architecture over closely related alternatives. However, the comparison with ERNIE-RNA does not isolate the architectural contribution, since the models differ in backbone design, pretraining data, optimization, and implementation. Although OptimRNA achieves better zero-shot results under the authors' unified nucleotide-dependency probe, the improvement over ERNIE-RNA is relatively modest, particularly in F1, and its statistical robustness is not established. A more controlled comparison with ERNIE-style attention propagation under a matched backbone and training recipe would substantially strengthen the evidence for the proposed architectural contribution.
W2. Insufficient evidence that a learned, persistent pair state is necessary over simpler structural-prior injection.
The pair state is initialized using a handcrafted structural prior that identifies candidate stems containing at least four consecutive Watson–Crick complementary pairs: Pij0=αΠij. This initialization incorporates biologically meaningful pairing information before learning. The state is subsequently refined through learned low-rank updates, antidiagonal propagation, and cross-layer accumulation.
The key question is whether the improvements specifically arise from learning and persistently accumulating pairwise information, rather than from introducing structural priors into attention or learning independent pairwise biases at individual layers.
The existing ablations compare a token-only baseline, a pair-state model without antidiagonal propagation, and the complete model. Although these experiments are useful, they do not sufficiently isolate the contributions of the initial structural prior, learned pairwise updates, and persistent cross-layer accumulation. In particular, three controls are missing:
- Fixed-prior baseline: A matched Transformer that uses the initial structural prior P0 as an attention bias without learned pairwise updates.
- Non-persistent pairwise baseline: A model that computes learned pairwise attention biases independently at each layer, without maintaining an accumulated pair state.
- Zero-initialization baseline: A model retaining the learnable persistent pair pathway but initializing P0=0.
Figure 4(a) provides evidence that the learned pair state becomes more structurally informative than its initialization, particularly in later layers. However, this does not establish that persistent accumulation is necessary or superior to simpler alternatives.
Furthermore, the substantial improvement obtained by supplying the pair state directly to the prediction head on ArchiveII raises a related question: how much of this gain requires a learned pair representation, rather than simply supplying the handcrafted structural prior as an additional input channel?
Since persistent pair-state refinement is a central contribution of the paper, I consider these missing comparisons a significant limitation.
W3. Insufficient specification of pair-prior initialization during MLM pretraining and potential information leakage.
The initial pair state is constructed from sequence-dependent nucleotide complementarity constraints, while MLM pretraining requires reconstructing corrupted nucleotide identities. However, the manuscript does not clearly specify whether P0 is constructed from the original sequence before masking or from the corrupted sequence after masking.
This distinction is important. If P0 is computed from the original sequence before MLM corruption, it may reveal information about the nucleotide identities that the model is supposed to predict, creating an unintended information pathway through pair-biased attention. The concern is particularly subtle because the initialization depends on runs of at least four consecutive complementary pairs, rather than only on the nucleotide identities at the two endpoints of each pair-state entry. Consequently, even removing pair-state rows and columns associated with masked positions might not eliminate indirect information leakage through neighboring candidate-stem entries.
If P0 is constructed entirely from the corrupted input sequence, this particular leakage concern can be avoided, provided the original nucleotide identities are not used elsewhere in constructing the prior. However, the resulting structural prior would be incomplete around masked positions, which may influence the model's behavior during pretraining.
I am not claiming that information leakage occurs. Rather, the current methodological description is insufficient to rule it out. Given that the model's structural prior is sequence-dependent and directly influences the MLM computation, the authors should explicitly clarify the order of masking and prior construction, including the handling of masked and randomly replaced positions. Pseudocode or a corresponding implementation reference would be helpful.
W4. Incomplete empirical comparison with ERNIE-RNA and insufficient evaluation of generalization.
(a) Zero-shot structure prediction.
The paper evaluates RNA language models using a unified nucleotide-dependency probe, which is useful for comparing representations under a common evaluation procedure. Under this protocol, OptimRNA-88M achieves better results than ERNIE-RNA on bpRNA TS0, with PR-AUC values of 0.5714 versus 0.5566 and F1 scores of 52.5% versus 51.7%.
However, ERNIE-RNA's original work also demonstrates a native zero-shot structure-prediction approach that directly interprets its learned attention maps. The original ERNIE-RNA paper reports a binary zero-shot F1 of 0.549 on bpRNA-1m using this approach. This result is not directly comparable with OptimRNA's zero-shot F1 because the evaluation metrics, prediction procedures, and decoding protocols differ. Nevertheless, it highlights an important distinction: the current experiments demonstrate an advantage for OptimRNA under the selected common probe, but do not establish an advantage over ERNIE-RNA's native zero-shot structural readout.
A more complete comparison would evaluate the native pairwise representations of both architectures under harmonized metrics and decoding procedures, while explicitly accounting for any head-selection or threshold-selection steps.
(b) Fine-tuned structure prediction.
Despite the proposed architectural improvements, OptimRNA does not outperform ERNIE-RNA on the main fine-tuned bpRNA TS0 benchmark. As shown in Table 1, ERNIE-RNA achieves a shift-tolerant F1 of 79.1% when its released fine-tuned checkpoint is re-evaluated under the authors' protocol, whereas OptimRNA-88M achieves 77.6%, including when the pair state is directly provided to the prediction head.
I recognize that these results are not a fully controlled architectural comparison, since ERNIE-RNA uses its own attention-map prediction head and the fine-tuning procedures differ. Nevertheless, the results do not demonstrate a consistent downstream performance advantage for OptimRNA over its closest architectural competitor.
Additionally, Appendix D.1 states that the authors' bpRNA fine-tuning results are obtained from a single seed. Reporting variability across fine-tuning runs, or confidence intervals for key performance differences, would help assess the robustness of the comparisons.
(c) Missing evaluation on challenging out-of-distribution benchmarks.
ERNIE-RNA evaluates secondary-structure prediction on several challenging benchmarks beyond bpRNA TS0, including RIVAS TestSetB, bpRNA-new, and RNA3DB-2D. These datasets provide complementary tests of generalization. RIVAS TestSetB evaluates transfer under different RNA family compositions, bpRNA-new tests generalization to RNA families absent from the supervised training benchmark, and RNA3DB-2D uses structurally dissimilar data splits to assess generalization beyond familiar structural patterns.
OptimRNA primarily evaluates generalization through ArchiveII leave-one-family-out experiments. While this is a valuable evaluation of cross-family transfer, it does not fully characterize performance on independent datasets with different family compositions or structural distributions.
Given that the proposed pair-state mechanism is intended to improve transferable structural representations, evaluation on these established benchmarks would considerably strengthen the paper. Without such results, it remains unclear whether the reported advantages persist under more challenging distribution shifts, particularly on datasets where ERNIE-RNA has already demonstrated strong performance.
Answer to Reviewer Lnso
We thank the reviewer for their positive assessment of our work.
Q1
Thank you for this suggestion. Following your request and Reviewer j4fN's complementary proposals, we substantially expanded the architectural ablations. All variants use the same 34M backbone, pretraining data, and training recipe. We report zero-shot secondary-structure prediction AUPRC (%) on bpRNA TS0, bpRNA-new, Rivas TestSetB and RNA3DB-2D using our common nucleotide-dependency readout, without fitting a downstream predictor. We add the difference from the full OptimRNA model in parentheses, and the statistical significance from Holm-adjusted, two-sided paired Wilcoxon tests (* p<0.05, ** p<0.01, *** p<0.001).
Zero initialization within OptimRNA does not significantly reduce performance on TS0, slightly improves it on bpRNA-new and worsens it on the Rivas TestSetB and RNA3DB-2D. It still retains a substantial advantage over the token-only backbone. Compared with the ablation of the antidiagonal convolution, the effect of removing the initial pair state is more dataset-dependent and less conclusive.
The full OptimRNA model achieves the highest average observed performance; initialization with prior knowledge contributes inconsistent and, on average, smaller improvements compared with the pair state itself and the antidiagonal convolution.
Q2
Thank you for raising the distinction between parameter efficiency and training-compute efficiency. Our controlled comparisons fix the pretraining recipe and number of gradient updates, allowing us to assess the benefit of the structural pathway at comparable data exposure. They do not establish an optimal allocation of a fixed FLOP budget, and we will clarify this distinction. Among the configurations evaluated, OptimRNA-34M provides a particularly favorable training-cost–performance trade-off. Increasing model size to 88M yields smaller structural-prediction gains than introducing the pair pathway into the 34M backbone. These observations support the smaller structure-aware model as a promising configuration under limited resources, although they do not determine whether a token-only backbone trained longer could close the gap.
Our additional experiments requested by Reviewer j4fN compare OptimRNA-88M with ERNIE-RNA retrained using the same recipe and approximately 202k updates. Under the validation-selected single-attention-map readout, OptimRNA achieves AUPRCs of 0.565 versus 0.552 on TS0 and 0.545 versus 0.486 on bpRNA-new, with comparable performance on Rivas TestSetB and higher performance on RNA3DB. The published ERNIE-RNA checkpoint reaches 0.585 and 0.614, respectively, but its reported pretraining required 20 days on 24 V100 GPUs, compared with approximately 40 hours on one H100 for OptimRNA-88M and 90 hours for our retrained ERNIE-RNA. These hardware-dependent figures are not a FLOP-normalized comparison, but illustrate the substantial difference in training resources. The published checkpoint’s stronger performance motivates investigating longer pretraining, although differences between recipes prevent attributing this gap to training duration alone. We are therefore extending OptimRNA-88M pretraining from approximately 202k to 606k gradient steps and will report the results when available. This experiment will assess whether the larger model benefits from additional training exposure. A comprehensive comparison of how best to allocate equal compute across architecture, model size, and training duration remains beyond the present evaluation; we will accordingly limit our efficiency claims to the parameter counts and measured computational costs supported by our experiments.
Q3
We include the baselines and the ablation on the antidiagonal convolution in our reply to Q1, with the statistical significance of the difference in performance compared with the full OptimRNA model. Removing the antidiagonal convolution significantly hurts performance across all datasets in the zero-shot setting, as does removing the pair state entirely. We will add statistical significance in the ablations in the paper.