- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies interactions between RNAs, specifically predicting the existence of interactions and the location of interactions between snoRNAs and their target RNAs. There are two major contributions: 1. propose SnoBench, an experimentally grounded benchmarking framework for both controlled candidate ranking (rank the interaction windows within a target RNA) and full-sequence target identification with site location (rank target RNAs according to their interaction probability with snoRNA and predicts site of interaction for each target RNA) tasks; 2. a novel partner-conditioned interaction model for two tasks by adding a learned signed nucleotide-pair attention bias and granularity-match dual heads. The two major contributions are both well-motivated and leads to conclusions that 1. strong discrimination among local candidate windows doesn't necessarily transfer to complete sequences; 2. target identification and site localization need to learn jointly.
Critical strengths
This paper proposes a more realistic benchmarking setup for RNA-RNA interaction prediction models when both the targets and its interaction sites are unknown. The new benchmarking and the conclusion from the benchmarking are impactful for this field to motivate new methods that can be used in practice. The paper also proposes a new methods for predicting targets and sites at the same time. Moreover, the paper is well written, especially the benchmarking part.
Critical weaknesses
Although the benchmarking part is well written, the partner-conditioned snoRNA interaction model section is less well written with several missing details (see my questions).
In the experimental section, some of the standard deviation are missing which makes it hard to justify the significance of the conclusions.
Questions for authors
- It seems that the partner-conditioned snoRNA interaction model takes a complete snoRNA sequence q and a candidate window w on target t (section 4.1). It is not clear how this model is applied on the whole sequence.
- Is the signed nucleotide-pair bias added to the trainable lookup table (B in eq.4) or is it an initialization of it? If the bias is only an initialization, how would it affect the results when optimization is converged?
- Where are H^w and H^q from? Do they have the same length as the window size and the query? What does \Omega_k mean in eq.6?
- In experimental results, the standard deviation represents variances across different runs. However, it would be more informative to show the variance across different train-test splits. Could the author show the variances of all results across different splits to show the statistical significance of the proposed methods?
6 comments
[2/2]
Q4. Standard deviations in full-sequence evaluation and variance across data splits (Addresses W2)
1. Complete 3-seed standard deviations for full-sequence N100 evaluation (W2):
We have completed full N100 evaluations across three training seeds for retrained RIME (NT), matched-backbone ERNIE-RIME, No-partner (Ours w/o partner conditioning), and Ours across the entire 986-pair cohort.
In controlled ranking, 3-seed MRR is 0.3982 ± 0.0044 for Ours, 0.3843 ± 0.0071 for No-partner, 0.3729 ± 0.0024 for ERNIE-RIME, and 0.3688 ± 0.0028 for RIME (NT). Complete 3-seed N100 full-sequence results (mean ± SD; recall in %) are:
| N100 metric | RIME (NT) | ERNIE-RIME | No-partner | Ours (Full) |
|---|---|---|---|---|
| Target R@1 | 2.54 ± 0.18 | 6.09 ± 0.37 | 5.19 ± 1.28 | 6.30 ± 0.33 |
| Target R@5 | 10.85 ± 0.46 | 11.73 ± 1.93 | 14.82 ± 1.41 | 14.52 ± 2.33 |
| Target MRR | 0.0915 ± 0.0012 | 0.1139 ± 0.0051 | 0.1210 ± 0.0036 | 0.1255 ± 0.0044 |
| Center Localization R@1 | 10.99 ± 1.02 | 13.83 ± 0.36 | 13.22 ± 0.93 | 14.36 ± 1.26 |
| Coverage Localization R@1 | 9.36 ± 0.77 | 12.37 ± 0.20 | 12.68 ± 0.97 | 13.37 ± 1.10 |
| Center Joint R@1 | 0.81 ± 0.35 | 4.60 ± 0.41 | 3.72 ± 1.13 | 4.81 ± 0.40 |
| Center Joint R@5 | 2.43 ± 0.37 | 6.22 ± 0.56 | 6.60 ± 0.60 | 7.11 ± 0.55 |
| Coverage Joint R@1 | 0.81 ± 0.35 | 4.19 ± 0.52 | 3.45 ± 1.24 | 4.45 ± 0.43 |
| Coverage Joint R@5 | 2.27 ± 0.26 | 5.85 ± 0.69 | 6.28 ± 0.53 | 6.75 ± 0.50 |
Across three seeds, Ours achieves higher three-seed mean performance than both RIME (NT) (nearly tripling Joint R@5 from 2.43%/2.27% to 7.11%/6.75%) and matched-backbone ERNIE-RIME on all reported endpoints.
2. Sampling uncertainty and consistency across query subsets:
SnoBench uses a fixed target-component-disjoint split and shared N100 candidate lists to standardize comparisons across methods (Appendices A.1 and A.3). Under this protocol, three training seeds quantify training variability, target-component-clustered bootstrap quantifies sampling uncertainty over held-out targets, and Query-Macro and stratified evaluations assess consistency across query weightings and subsets:
Updated target-component-clustered bootstrap: Using paired target-component bootstrap on seed-averaged per-pair metrics, the 95% confidence intervals for Ours minus matched-backbone ERNIE-RIME exclude zero on Target MRR (+0.0116 [+0.0041, +0.0195]), Target R@5 (+2.79 [+0.97, +4.96] pp), Center Joint R@5 (+0.89 [+0.18, +1.59] pp), and Coverage Joint R@5 (+0.90 [+0.20, +1.65] pp); against RIME (NT), the clustered 95% CIs on Center and Coverage Joint R@5 are +4.67 [+0.71, +10.36] pp and +4.49 [+0.53, +10.21] pp.
Consistency across query subsets: Evaluating 3-seed N100 performance under equal query weighting (Query-Macro across all 46 queries) and stratified by Non-U3 vs. U3 subsets confirms positive mean gains across all regimes:
| Evaluation regime | Target MRR: ERNIE-RIME | Target MRR: Ours | Target R@5 (%): ERNIE-RIME | Target R@5 (%): Ours | Coverage Joint R@5 (%): ERNIE-RIME | Coverage Joint R@5 (%): Ours |
|---|---|---|---|---|---|---|
| Query-Macro (46 queries) | 0.1490 ± 0.0092 | 0.1787 ± 0.0195 | 17.43 ± 7.92 | 29.94 ± 7.81 | 9.13 ± 1.28 | 11.61 ± 0.46 |
| Non-U3 (186 pairs) | 0.1646 ± 0.0008 | 0.1856 ± 0.0175 | 17.92 ± 5.49 | 25.45 ± 5.95 | 11.11 ± 1.24 | 13.35 ± 0.56 |
| U3 (800 pairs) | 0.1021 ± 0.0061 | 0.1115 ± 0.0029 | 10.29 ± 1.25 | 11.98 ± 1.60 | 4.62 ± 0.57 | 5.22 ± 0.51 |
We will add the inference details to Section 4 and the complete three-seed statistics to Table 2 in the revised manuscript, and we would appreciate your reassessment with these additions.
Official Comment
Thank you for your effort in rebuttal and thank you for incorporating the clarifications! About the standard deviations of the performance, could you please report the statistical significance level of the proposed method to the baseline methods, such as using a t-test?
[1/2]
We sincerely thank you for recognizing the practical impact of SnoBench and our partner-conditioned model, as well as the clarity of our benchmark formulation. We address each of your questions (Q1–Q4) and the concerns in W1–W2 below, and we will incorporate these clarifications and complete three-seed standard deviations into Sections 4–6 of the revised manuscript. All newly reported ERNIE-based results use the corrected native attention-bias layout (see our response to Reviewer f6GR, Q2) and validation-selected checkpoints across three training seeds.
Q1. How the window-level model is applied to complete target sequences (Addresses W1)
The same candidate-local scorer connects SnoBench's two settings: it ranks supplied windows in controlled evaluation and scans complete candidate targets to produce both target rankings and interaction sites (Appendix C.1):
Candidate window generation: Given a complete candidate target t of length L, we generate candidate windows Iₜ across widths w ∈ {21, 42, 70} nt (for w ≤ L) advancing by stride ⌊w/2⌋, with an additional window starting at L − w when needed to cover the target end.
Contextual scoring: Each candidate window k ∈ Iₜ is taken together with up to 64 nt of flanking sequence on both sides (clipped at target boundaries) to form the target context, which is paired with the full snoRNA q and scored by the model to produce zₖ = aₖ + ℓₖ (Eq. 7).
Target-level reduction and site localization (Eq. 10): Across all N = 100 complete candidate targets, each target t is ranked by its maximum window score Sₜ = max of zₖ over k ∈ Iₜ, and the window achieving this maximum (ŵₜ = argmax of zₖ over k ∈ Iₜ) is output as its predicted interaction site.
Q2. Parameterization of the signed nucleotide-pair bias and its behavior after training (Addresses W1)
1. Initialization vs. additive term: In Eq. (4), the cross-attention bias for head h is αₕB⁽ʰ⁾, where B⁽ʰ⁾ᵢⱼ = Tₕ[xᵢ, yⱼ]. The canonical pairing prior (Watson–Crick = 3.0, G–U/U–G wobble = 1.5, others = 0) initializes each head's nucleotide-pair lookup table Tₕ, while the gates αₕ are initialized to +1, −1, or 0 (four heads per group). Both Tₕ and αₕ remain trainable.
2. Behavior of the effective bias in the trained models: In the submission's Table 3, "w/o pair prior" removes the additive pairing-bias term αₕB⁽ʰ⁾ from cross-attention (Eq. 4; Appendix B.4), lowering controlled MRR. In the corrected three-seed models, the relative Frobenius-norm change across the concatenated effective bias tables (αₕTₕ) is below 0.185%. Positive- and negative-group gates retain their initialized signs, and their effective tables retain their respective initialized pairing orderings; neutral-group gates remain near zero (max |αₕ| ≤ 0.0045). These results indicate that the signed structural prior is preserved after training.
Q3. Origin and lengths of Hq and Hw, and meaning of Ωₖ (Addresses W1)
Origin and lengths of Hq and Hw: In Section 4.1, the frozen encoder produces nucleotide representations Xq=E(q) for the full snoRNA q and Xc=E(t~w) for the target context (the candidate window w plus flanking sequence on both sides). Both Xq and Xc are passed into the bidirectional cross-attention block (Section 4.2, Eq. 4):
Meaning of Ωₖ in Eq. (6): For candidate k, the 2D interaction map P (Eq. 5) and feature map Fₖ are indexed over snoRNA positions i ∈ {1, …, |q|} and window positions j ∈ {1, …, |wₖ|}. Because query and window lengths vary across instances, Ωₖ = {1, …, |q|} × {1, …, |wₖ|} denotes the set of valid, unpadded 2D nucleotide-pair positions (i, j) for candidate k (|Ωₖ| = |q|·|wₖ|). Pooling over Ωₖ in Eq. (6) (and Eq. 17) aggregates features strictly over valid, unpadded positions.
We will clarify the transition from (Xq, Xc) to (Hq, Hw), their lengths |q| and |w|, and the definition of Ωₖ in Sections 4.2–4.3.