- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes ReMatch, a training objective for RNA–protein interaction prediction that combines standard binary classification with bidirectional ranking over alternative pairings. Instead of treating all re-paired RNA–protein alternatives as confirmed negatives, ReMatch uses them as relative comparisons: for an observed interaction, the model is trained to score the observed pair above alternatives formed by replacing either the RNA or the protein. This allows plausible unverified alternatives to remain above the interaction boundary while still giving higher score to experimentally supported pairs. The method uses frozen RNA and protein language-model features with a trainable interaction head, then applies ranking directly to the classifier’s interaction log-odds.
Critical strengths
- The paper correctly highlights that many “negative” RNA–protein pairs are generated by re-pairing molecules and may simply be untested rather than truly non-interacting. Using these alternatives as comparisons, rather than hard negative labels, is a sensible way to reduce the risk of suppressing plausible interactions.
Critical weaknesses
-
The paper reports validation means rather than results on an untouched test set: within each fold, the checkpoint is selected by validation accuracy and then evaluated on that same validation split. This can introduce optimistic bias and makes the reported generalization performance less convincing.
-
The statistical evidence is also not very strong. Although the paper reports gains from adding ranking, the paired confidence intervals are wide. Because these intervals cross zero, interpret the reported gains as promising point estimates rather than firm, statistically significant improvements.
-
The baseline comparison is not fully matched. The reproduced baselines are reported as one run per setting, whereas ReMatch is reported as five-fold validation means. This makes it difficult to separate the effect of the proposed ranking objective from differences in training protocol, model selection, and fold averaging.
-
The methodological novelty is somewhat limited. The ranking loss is based on standard pairwise preference/ranking ideas such as Bradley–Terry, RankNet, and BPR, and the in-batch cross-pair score matrix resembles common contrastive or retrieval-style training constructions. The main contribution is therefore not a fundamentally new ranking objective, but the application of this idea to RPI with unreliable negatives, the use of interaction log-odds, bidirectional RNA/protein replacement, and a careful set of controlled analyses.
-
The biological interpretation could also be clearer. The method assumes that experimentally observed RNA–protein pairs should be ranked above re-paired alternatives, but this preference mainly reflects available experimental evidence rather than guaranteed biological incompatibility of the alternatives. The distinction between true non-interactions, untested alternatives, hidden positives, and experimentally supported observations is dense and may be difficult for readers to follow.
-
There is also potential pretraining leakage that is not fully checked. The downstream splits avoid exact RNA/protein identity overlap, but the frozen RNA and protein language models may have seen validation-set sequences during pretraining. The paper notes that encoder-pretraining exposure would require a separate provenance check, so this remains an unresolved concern.
-
Several design choices may require tuning, including top-k hard alternative selection, ranking temperature, replacement direction, ranking schedule, and the score location used for supervision. Although the paper includes useful ablations, it still does not fully establish practical robustness to these choices.
-
On the Cas v2 transfer set, all models stay close to chance, and ReMatch only marginally beats the strongest baseline, with no uncertainty reported. This analysis also uses the earlier ReMatch + MI checkpoint rather than the final model, and it has no BCE-only control. It therefore does not show whether the ranking objective helps cross-domain transfer.
Questions for authors
My questions for the authors largely follow from the weaknesses discussed above. I would especially appreciate clarification on the lack of an independent test set, the statistical reliability of the reported gains, the mismatch between baseline and ReMatch evaluation protocols, the degree of novelty beyond standard pairwise ranking losses, the possibility of pretraining exposure to validation sequences through frozen encoders, and the biological interpretation of treating observed pairs as preferred over unverified alternatives.
Thank you for helping us strengthen the study. We conducted 135 evaluations across seven predictors and two ReMatch variants, plus a full hyperparameter scan and tests of pretraining overlap and Cas transfer without MI. We include one table for space, and full results and procedures will accompany the revision.
Response to Q1:
We appreciate your suggestion to make evaluation more rigorous. Our original protocol followed the RPI baselines for comparability. We now use separate training, validation, and test data in each of five folds, following broader ML practice. We reserve 15% of each training partition for validation and checkpoint selection, then evaluate on the original held-out fold. These are revised runs on existing data. ReMatch's gains persist, and the clearer protocol should also help future RPI comparisons.
Table 1. Five-fold mean +/- standard deviation. Classification denotes BCE training of the same network without ranking.
Response to Q2:
Your careful assessment helps us state the evidence more precisely. We will describe the original CW gains as promising point estimates, since their paired intervals cross zero. We reran four matched controls with separate checkpoint selection: BCE, BCE-H, BCE+Rank, and BCE-H+Rank.
BCE+Rank improves mean accuracy and AUROC over BCE on all three benchmarks, with accuracy gains of 8.5 and 3.6 percentage points on CW30_50 and CW60_80. Hard-negative classification improves CW AUROC but lowers NPInter performance, while adding it to ranking gives no consistent further benefit. This supports learning preferences from sampled alternatives while retaining standard BCE for classification. We will include these controls in the revision so readers can assess the size and consistency of the gains.
Response to Q3:
Thank you for emphasizing fair comparisons. We reran all six existing predictors and report five-fold means and variation. ReMatch achieves the highest mean accuracy and AUROC on both CW benchmarks while remaining competitive on NPInter. The matched BCE control further shows that adding ranking improves the same network under this evaluation protocol.
Response to Q4:
Your comment helped us clarify the contribution. We use established ranking losses with alternatives formed by replacing either molecule. With BCE retained under the revised protocol, we compared cosine similarity and interaction log-odds. CW30_50 accuracy is 0.639 versus 0.721 for individual comparisons and 0.626 versus 0.723 for joint comparisons. Both favor supervising the score used for prediction.
Response to Q5:
We appreciate the distinction you raise: an observed interaction need not bind more strongly than an unobserved one. To test how this affects learning, we withheld known interactions and used them as alternatives. When withheld positives form 10.65% of selected alternatives, ReMatch retains 22.7% of the withheld set above the prediction threshold, versus 11.3% when BCE and ranking are retained but alternatives also receive negative labels. Retention is also higher at the other two rates. This supports using comparisons to preserve plausible interactions.
Response to Q6:
Thank you for encouraging this direct check. Unlabeled sequence exposure does not itself imply leakage of interaction labels. We split each test set into disjoint subsets with and without exact RNA matches to the checked RiNALMo pretraining source. On CW30_50 and CW60_80, ReMatch's AUROC is 0.787 and 0.750 with overlap, but 0.873 and 0.849 without overlap. Its predictive value thus extends beyond sequences present in that source.
Response to Q7:
We appreciate the practical concern. We tested every combination in our grid of top-k, temperature, replacement direction, and ranking schedule, alongside score-location controls. The results support k=5, temperature 0.25, bidirectional replacement, and two warm-up epochs. CW30_50 mean AUROC ranges from 0.819 to 0.832 across candidate counts, suggesting flexibility in this choice. The full results will provide a reproducible starting point and show where tuning matters.
Response to Q8:
Thank you for identifying the missing transfer control. We compared final ReMatch without MI against BCE-only on Cas v2. ReMatch has higher point estimates on both reported metrics. Although absolute scores remain modest, this is encouraging evidence that ranking benefits the final model without relying on MI. We will add these results to the revision.