- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SIRGE, an evaluator for RNA 3D structure candidates. It conditions an SE(3)-invariant geometric encoder (EquiformerV2 → nucleotide frames → IPA) on attention maps from three selected heads of a frozen RiNALMo, injected through FiLM. Training adds a tertiary-contact recovery objective to the ranking loss. SIRGE beats six baselines on CLS-1 (Kendall-τ 0.270 vs. 0.159) and on Gen-OOD (Hit@1 0.246 vs. 0.135). An interpretability analysis shows that the selected heads carry tertiary-contact signal beyond base pairing.
Critical strengths
-
The motivation is clear and grounded in biology: candidates for long RNAs often share stems but differ in helix packing, and the sequence is always available at scoring time.
-
Appendix B uses careful controls (separation binning, matching on secondary-structure distance, shuffled nulls). The Perm ablation shows that the gain depends on the prior matching the sequence.
Critical weaknesses
-
CLS-1 has only 21 RNAs, and the ± values reflect only three training seeds. The long-RNA claim rests on 11 RNAs. The “non-additive” ablation gap (0.012) is within seed noise. RNA-level paired bootstrap or significance tests are needed.
-
SIRGE-base, with no prior and no contact objective, already reaches 0.217 and beats every baseline. About half of the headline “70%” comes from the architecture. The title and abstract overstate the role of alignment, and SIRGE-base should appear in all main tables, including Gen-OOD.
-
With six candidates, random Hit@1 is about 0.167, and every baseline scores below that, with negative τ. Simple strong baselines are missing: always picking AF3, predictor pLDDT, and consensus/clustering. Most references are bound conformations from complexes.
-
There is no comparison with a non-LM prior, such as ViennaRNA/EternaFold base-pair probabilities or RiNALMo’s secondary-structure head. Without it, the LM-specific contribution is not established. Statistical potentials (rsRNASP, cgRNASP, etc.) and Liu et al. (2026) are not compared.
-
SIRGE may simply reward compactness or contact count. Contacts shared with the native should be separated from spurious ones, controlling for radius of gyration, across all long RNAs.
Questions for authors
How does SIRGE do with a BPP prior in place of the LM attention maps?
What Hit@1 do “always AF3”, pLDDT and consensus baselines reach on Gen-OOD, and how does SIRGE-base do there?
Can you provide RNA-level significance tests?
Do the conclusions hold under TM-score or lDDT?
7 comments
Rebuttal 2/5
Question 2: What Hit@1 do “always AF3”, pLDDT and consensus baselines reach on Gen-OOD, and how does SIRGE-base do there?
We thank the reviewer for suggesting these simple selection baselines. We computed them on Gen-OOD and report the results below.
| Selection rule | Hit@1 |
|---|---|
| always AlphaFold 3 | 10/84 = 11.9 % |
| highest pLDDT | 11/84 = 13.1 % |
| consensus | 14/84 = 16.7 % |
| random choice among the six candidates (expectation) | 16.7 % |
| SIRGE-base | 21.8 % |
| SIRGE | 24.6 % |
The pLDDT rule selects the candidate with the highest mean pLDDT. The consensus rule selects the candidate with the lowest mean RMSD to the other five, i.e., the most central structure among the six predictions. Replacing RMSD with lDDT as the agreement measure gives 9/84 = 10.7%.
All three heuristics are at or below random selection, whereas SIRGE reaches 24.6% Hit@1. This directly addresses the concern that a simple fixed rule might suffice: no simple fixed rule can provide a reliable shortcut for identifying the best candidate across targets. Instead, generator quality is target-dependent, highlighting the need for an evaluator that makes target-specific selections.
Rebuttal 3/5
Question 3: Can you provide RNA-level significance tests?
We thank the reviewer for the question and report below the requested analysis.
All statistics below take the RNA as the unit of analysis: each RNA contributes its Top-3 ordering accuracy, averaged over the three seeds, confidence intervals (CI) are 95% percentile bootstrap intervals over RNAs (20,000 resamples), and tests are two-sided Wilcoxon signed-rank tests on the paired per-RNA differences. On CLS-1 SIRGE's Top-3 accuracy has a 95% interval of [0.270, 0.349], above the mean of every baseline (0.212 to 0.265), and its lift over each baseline excludes zero:
| Comparison | Mean Improvement (↑) | 95% CI | RNAs where SIRGE improves | p-value |
|---|---|---|---|---|
| SIRGE vs. HomRank | +0.043 | [+0.024, +0.063] | 16/21 | < 0.001 |
| SIRGE vs. lociPARSE | +0.055 | [+0.023, +0.088] | 16/21 | 0.004 |
| SIRGE vs. ARES | +0.052 | [+0.022, +0.085] | 16/21 | 0.003 |
| SIRGE vs. RNA3DCNN | +0.093 | [+0.058, +0.132] | 18/21 | < 0.001 |
| SIRGE vs. EquiRNA | +0.076 | [+0.042, +0.112] | 17/21 | < 0.001 |
| SIRGE vs. PAMNet | +0.097 | [+0.057, +0.141] | 18/21 | < 0.001 |
On Gen-OOD SIRGE's Top-3 interval is [0.246, 0.393], above the mean of every baseline (0.179 to 0.206), and its lift over each baseline excludes zero:
| Comparison | Mean Improvement (↑) | 95% CI | RNAs where SIRGE improves (ties) | p-value |
|---|---|---|---|---|
| SIRGE vs. HomRank | +0.127 | [+0.044, +0.210] | 33/84 (38) | 0.012 |
| SIRGE vs. lociPARSE | +0.111 | [+0.016, +0.206] | 32/84 (38) | 0.056 |
| SIRGE vs. ARES | +0.139 | [+0.044, +0.234] | 35/84 (31) | 0.007 |
| SIRGE vs. RNA3DCNN | +0.139 | [+0.044, +0.234] | 36/84 (35) | 0.016 |
| SIRGE vs. EquiRNA | +0.144 | [+0.026, +0.255] | 25/51 (17) | 0.029 |
| SIRGE vs. PAMNet | +0.139 | [+0.048, +0.230] | 37/84 (29) | 0.004 |
EquiRNA's published neighbourhood size (26) cannot build a graph for RNAs of 26 nt or fewer; its row is paired on the 51 Gen-OOD RNAs it scores (28–135 nt).
Rebuttal 1/5
We thank the reviewer for their careful and thorough review. We address each question separately
Question 1: how does SIRGE do with a BPP prior in place of the LM attention maps?
We thank the reviewer for the question. To answer it, we trained SIRGE with a BPP prior from the two distinct families of secondary-structure predictor that the reviewer mentioned: ViennaRNA (Lorenz et al., 2011), the classical thermodynamic method, and RiNALMo-SS, the RiNALMo language model fine-tuned for base-pair prediction (Penić et al., 2025). We apply the base pairing maps from these two methods to replace the attention maps conditioning in our SIRGE. See the updated Appendix for more details. As the table below shows, both are stronger base-pairing priors than the attention maps.
Both BPP priors return SIRGE to the level of no prior at all, and the two predictors, despite their different methodologies, give the same result. The deficit is concentrated where SIRGE's gain is concentrated, on the long RNAs (Figure 3): on the 11 RNAs of 150 nucleotides or more, SIRGE reaches a Kendall-τ of 0.238, against 0.160 with the ViennaRNA prior and 0.171 with the RiNALMo-SS prior. This is the regime in which a prior without tertiary information has the most to lose: longer RNAs contain more stems and admit more ways of arranging them, so their packing is the harder part of the problem. The deficit is not a matter of predictor accuracy, since restricting to the RNAs on which each predictor recovers most of the native pairs does not close the gap. Nor is it a matter of the maps being ignored: their FiLM gates grow as large as SIRGE's on the attention maps. The maps are used, and they are not informative about packing.
Sequence changes that preserve every base pair still alter how the helices pack (Yesselman et al., 2019). Figure 5a,b supports this distinction: the stem contacts stay constant across the quality range while the tertiary contacts fall as RMSD grows, so good and bad candidates often share most of the same base pairs and differ in packing. Since the prior is shared by all candidates of an RNA and acts only through its interaction with their geometry (Section 3.1), a map that agrees equally well with every candidate's stems and holds no expectation about which distant nucleotides should touch cannot favour the correct packing. The selected attention heads carry that expectation: Appendix B shows their tertiary-contact AUC stays above chance at all separations of 24 nucleotides and more, after matching on sequence separation and secondary-structure distance.
The attention maps therefore help not through the secondary structure they encode but through the sequence-specific packing information they carry beyond it, which the contact objective trains SIRGE to recover and use as its reference. We have added both ablations to Table 4 and their construction to Appendix A.4.