acceptodds
Under review as a conference paper at ICLR 2027

Agentic Orchestration of Heterogeneous Predictors for RNA Binding Pocket Discovery

Abstract

Protein surfaces that engage RNA are targeted today by four disjoint families of tools: end to end complex structure prediction, surface pocket detection, binding residue classification, and docking. Each encodes a different inductive bias and is validated under its own protocol, so a practitioner facing a new target has no principled basis for choosing among them. On a structurally disjoint benchmark curated from 1,809 experimental complexes, no single tool is universally best, per target performance spreads more widely *within* a family than *between* families, and the winner is partly predictable from target properties. Per target tool choice, rather than another monolithic model, is therefore the operative design variable. RiboSeer acts on this by separating planning from arithmetic: a language model profiles each pair and selects from a 15 tool library subject to a mandatory core covering all four paradigms, a learned gradient boosted fusion combines per residue scores, cross tool agreement, neighborhood context, and the agent profile into a calibrated binding probability, and an exponential moving average over per category reliability closes the loop across samples. On 107 held out targets separated from training by TM score, RiboSeer reaches Pearson of against for the best single tool and for a weighted mean over the same tools, with AUROC . The margin holds across four splits, four contact cutoffs, and strata by RNA length, domain family, and difficulty. Ablations attribute the gain: learned fusion over a paradigm diverse pool supplies most of it, while the language model contributes Pearson over a bandit only selector.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
RNA Binding Pocket, Binding Site Prediction, Protein-RNA Interaction, LLM Agents, Tool Selection, Learned Fusion, Structural Bioinformatics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_A3CNOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RiboSeer, an agentic framework for RNA-binding pocket prediction. The framework integrates 15 existing tools across four categories: complex structure prediction, pocket detection, binding residue prediction, and molecular docking. An LLM-based agent analyzes target characteristics and selects appropriate tools, and a LightGBM-based fusion model combines their predictions to produce residue-level binding probabilities. RiboSeer is evaluated on a benchmark of 107 held-out targets. It achieves a Pearson correlation of 0.593 and an AUROC of 0.899, outperforming the best individual predictor (Pearson r = 0.439) and a weighted ensemble baseline (0.515). Ablation studies show that most improvements come from learned prediction fusion, while LLM-based orchestration provides an additional but relatively modest contribution (~7% improvement). Overall, the paper demonstrates the practical benefits of adaptively combining heterogeneous scientific prediction models.

Critical strengths
  1. The paper presents an interesting and practical idea: dynamically selecting and combining different models based on the characteristics of each problem, rather than relying on a single model. Given the complementary strengths and limitations of existing predictors, I believe this is a meaningful and potentially generalizable direction for scientific applications.

  2. The proposed framework achieves good empirical performance. On 107 held-out targets, RiboSeer achieves a Pearson correlation of 0.593, outperforming the best individual predictor Boltz-2 (0.439), as well as a simple weighted ensemble method (0.515). These results demonstrate the practical value of combining heterogeneous predictors.

Critical weaknesses
  1. While the overall framework is well-motivated, the technical contributions is relatively limited. The proposed method mainly integrates existing techniques, including LLM-based tool selection, bandit-based exploration, and LightGBM-based prediction fusion. The paper focuses more on system integration than on introducing fundamentally new algorithms or methodological advances.

  2. Although the paper emphasizes agentic orchestration, the actual contribution of the LLM agent is relatively modest. The ablation results show that adding the agent improves Pearson correlation from 0.554 to 0.593, corresponding to an approximately 7% relative improvement. Moreover, the agent primarily acts as a model router rather than performing more critical scientific reasoning or decision-making. This raises questions about the necessity of an LLM-based agent compared with simpler model selection strategies, particularly given that most of the performance gains come from learned prediction fusion. Additionally, the paper's presentation could be improved. Figure 1 is overly dense, and its text and visual elements are too small to read comfortably.

Questions for authors

The proposed framework seems potentially applicable to a broader range of scientific prediction tasks. Do the authors have plans to extend this approach to other tasks? More importantly, do the authors believe the current framework can be readily generalized to different scientific problems, or would substantial task-specific adaptations be required?

2 comments
AuthorsOpenReview ↗

Response to Reviewer A3CN (1/2)

Dear Reviewer A3CN,

Thank you for the review, and for seeing the per-target selection idea as a direction worth pursuing beyond this task. Your two criticisms both landed, and the revised PDF (changes marked in blue) acts on them rather than argues with them.


W1. Limited technical novelty

We agree that LLM tool selection, bandit utilities and LightGBM fusion are established, and we do not claim a new learning algorithm. What we claim, and now state that way in the contributions, is the integration and comparative evaluation: 15 predictors from four paradigms produce scores on different scales with different failure modes and cannot simply be averaged, so RiboSeer standardizes residue-level outputs through per-tool adapters, handles failed predictions, and fuses per-tool scores together with cross-tool agreement, sequence-neighbourhood context and a target profile.

The evidence that this is not a trivial wrapper is the gap between having the tools and using them: best single tool 0.4390.439, simple mean over the same 15 tools 0.4460.446, train-weighted mean 0.5150.515, learned fusion 0.593\mathbf{0.593}. On the 104 targets where Boltz-2 is valid, the paired gain over it is +0.155+0.155 (95% CI [0.121,0.188][0.121, 0.188], 10,000 PDB-grouped bootstrap resamples), which we added in revision so that the headline comparison is made on a common set of targets.

W2. Necessity of the LLM agent, and Figure 1

You and the other reviewers converged on the same point, so we tested it directly instead of re-interpreting the 0.554→0.5930.554 \rightarrow 0.593 ablation. On the same 107 targets (Table 29 in the revision):

Selection methodPearson rrΔ\Delta vs. Maestro (95% CI)
Maestro (LLM)0.59308—
Fixed seven-tool core0.59407+0.00099 [−0.00501, 0.00625]+0.00099\ [-0.00501,\ 0.00625]
Non-LLM KNN selector0.59031−0.00277 [−0.01166, 0.00476]-0.00277\ [-0.01166,\ 0.00476]
Random extra tools (30 runs)0.58823−0.00485 [−0.01083, 0.00069]-0.00485\ [-0.01083,\ 0.00069]

No interval excludes zero, so none of these comparisons supports an accuracy benefit specific to LLM reasoning. The original ablation could not isolate it either, because the mandatory core is present in both arms. The abstract, Sec. 2, Sec. 4.4 and the conclusion now say that learned fusion supplies the gain and that LLM selection is on par with a fixed core and a non-LLM selector; we present it as a usable interface rather than a source of accuracy. Your own reading, that the agent acts as a router, matches what the numbers show, and the paper now reads that way.

For completeness, the router is not arbitrary: its selected set contains the retrospectively best single tool for 86.0%86.0\% of targets against 80.4%80.4\% for the core alone, and the per-target effect runs both ways (on 8inf_u_x adding GraphBind lifts fused rr from 0.3070.307 to 0.4800.480; on 8cdr_NN_Bb the extras drop it from 0.0930.093 to −0.001-0.001). We report this as coverage behaviour, not as a demonstrated benefit.

Figure 1. Agreed, it is too dense. The revision adds an enlarged two-panel version in Appendix A.17 so the labels are legible now, and the final version will replace Fig. 1 with a simplified redraw with fewer boxes and larger type. The caption also corrects a panel label: the fusion is a regressor producing continuous scores, not a classifier (Sec. 3.7).

Two further corrections from this round may interest you, since they affect how the results should be read: the pocket-level evaluation now fixes pocket size from training data instead of from test labels, and with that correction RiboSeer leads Boltz-2 on every pocket metric (Table 3); and the headline reproduces exactly (107/107107/107 targets, mean 0.593080.59308) from one hashed frozen checkpoint, documented in Appendix A.13.


(continued in part 2)

AuthorsOpenReview ↗

Response to Reviewer A3CN (2/2)

(continued from part 1)

Q. Generalizing to other scientific tasks

Reusable: the adapter contract that standardizes incompatible predictor outputs, the fallback for failed tools, the feature construction over tool agreement and context, and the learned fusion. In our experience these are the parts that take the time and are the parts that transfer as a recipe.

Not reusable without work: a new task needs its own predictors and adapters, its own label definition, its own fusion features and training data, and, if online feedback is used, a task-appropriate surrogate reward. We have run preliminary extensions toward RNA–ligand, protein–ligand and protein–protein interface prediction, and each required task-specific tools and inputs; we do not present those as a controlled cross-task demonstration, so they are not in the paper.

The conclusion now states this explicitly: the framework is a reusable recipe for domains where incompatible predictor families coexist under separate protocols, not a zero-shot capability. We would rather scope it that way than imply generality we have not measured.


We think the revision is now an accurate account of where the value sits: a heterogeneous benchmark and a fusion result that reproduces exactly and holds up under paired tests and a corrected pocket protocol, with the orchestration component reported honestly as interface rather than mechanism. If a further experiment would help your assessment, we are glad to run it during the discussion period.

Thank you again for the constructive feedback. If the revised contribution statement and new comparisons address your concerns, we would appreciate it if you would consider raising your score.

Best regards,

Authors

Reviewer_MqbnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RiboSeer, a framework that integrates 15 heterogeneous predictors from four methodological paradigms for protein–RNA binding site prediction. It employs LLM-based target profiling and tool selection, LightGBM-based prediction fusion, and a label-free feedback mechanism for updating tool reliability.

On a benchmark of 332 representative targets, RiboSeer achieves a Pearson correlation of 0.593 on 107 held-out targets, outperforming the best individual predictor (0.439) and a weighted averaging ensemble (0.515). The authors further evaluate the framework through ablation studies, alternative data partitions, and stratified analyses.

While the integration of heterogeneous predictors yields promising results, the independent contribution of the proposed agentic orchestration remains insufficiently established, and several aspects of the experimental evaluation require clarification.

Critical strengths
  1. Practically valuable integration of heterogeneous predictors.

The paper unifies 15 predictors spanning complex structure prediction, surface pocket detection, binding residue classification, and molecular docking. The common evaluation framework and learned fusion approach provide a useful way to exploit complementary information across prediction paradigms.

  1. Promising empirical results with relatively extensive analyses.

RiboSeer demonstrates substantial improvements in residue-level prediction over individual predictors and conventional ensemble baselines. The experiments cover fusion strategies, feature ablations, tool contributions, alternative data partitions, and failure cases, providing meaningful evidence for the effectiveness of heterogeneous predictor fusion.

Critical weaknesses
  1. Insufficient evidence for the independent contribution of LLM-based orchestration.

The paper positions LLM-based tool selection as a central contribution, yet the reported gains appear to be largely driven by heterogeneous predictor fusion. The full model achieves a Pearson correlation of 0.593, compared with 0.554 for Bandit-only selection and 0.575 when all 15 predictors are executed with learned fusion. Moreover, seven tools are always included, leaving the LLM to select approximately two additional tools per target on average. Appendix Table 9 further shows that residue-level features alone achieve 0.589.

These results raise questions about the necessity of LLM reasoning. Crucial comparisons against a core-only baseline, random additional tool selection, and non-LLM instance-conditioned selectors under matched computational budgets are missing. The paper also lacks target-level evidence demonstrating how LLM reasoning leads to better tool choices. The abstract's claim that the best predictor is partly predictable from target characteristics is not directly quantified.

Consequently, the independent methodological contribution of the proposed agentic orchestration remains unclear.

  1. Experimental inconsistencies and insufficiently specified evaluation protocols weaken the main conclusions.

Several reported results are difficult to reconcile. The sample-count-weighted averages of RiboSeer's Pearson correlations in Appendix Tables 15–18 are approximately 0.587–0.588, rather than the 0.593 reported in Table 2. Section 4.5 also reports improvements of +0.075 and +0.071 for the temporal and tighter identity splits, whereas Appendix Table 20 indicates +0.114 and +0.111. These discrepancies require clarification regarding the checkpoints and evaluation protocols used.

The independence and fairness of the evaluation also warrant further examination. The framework incorporates externally pretrained predictors whose training data may overlap with the test targets. The manuscript does not clearly specify whether structural inputs to pocket detection and docking methods are experimentally determined bound structures, unbound structures, or independently predicted structures. Furthermore, individual predictors are evaluated on different numbers of valid targets, making paired comparisons on a common test subset desirable.

The proposed label-free feedback mechanism also lacks a clear definition of how the overall surrogate reward is attributed to individual tools. Finally, while residue-level performance improves substantially, Table 3 shows limited gains in three-dimensional pocket localization, with IoU and Top-1/Top-3 success rates failing to surpass the strongest baseline.

Together, these issues limit the reproducibility and interpretation of the reported results and the extent to which they support the paper's central claims.

Questions for authors
  1. Independent value of LLM orchestration

Could the authors compare MAESTRO against core-only, core-plus-random, and non-LLM instance-conditioned tool selection under matched computational budgets? Could target-level examples demonstrate when LLM reasoning leads to better tool choices? How predictable is the identity of the best tool from target characteristics?

  1. Consistency of experimental results.

Why do the weighted averages of Appendix Tables 15–18 differ from the headline Pearson correlation of 0.593? Could the authors also reconcile the improvement values in Section 4.5 and Appendix Table 20, and clarify whether different checkpoints or evaluation protocols were used?

  1. Evaluation independence and fairness.

What are the sources of the protein and RNA structures supplied to each predictor? Have potential overlaps between the underlying predictors' pretraining data and the test targets been assessed? Could the authors provide paired comparisons on common valid targets, with confidence intervals for the key improvements?

  1. Validity of adaptive feedback.

How is the overall surrogate reward decomposed into tool-specific reliability updates? Has its correlation with actual tool-level prediction quality been validated? Given the modest improvement from MEMORY in Appendix Table 14, what evidence establishes the benefit of this mechanism?

  1. Binding pocket localization.

How do the authors explain the discrepancy between residue-level improvements and the limited pocket-level gains in Table 3? Could they clarify the pocket extraction procedure and evaluate localization without relying on the ground-truth number of contacting residues?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Mqbn (1/3)

Dear Reviewer Mqbn,

Thank you for the careful reading, and in particular for catching both numerical inconsistencies. Both were real, both are now traced and fixed, and the revised PDF marks every change in blue. A short map of what changed:

Your concernWhat we did
LLM orchestration not isolatedAdded core-only, core-plus-random and non-LLM KNN selectors with paired CIs (Table 29); claim rewritten (Sec. 4.4, abstract, conclusion)
Tables 15–18 vs. Table 2Cause identified; tables regenerated from the formal checkpoint (App. A.13)
Sec. 4.5 vs. Table 20Manuscript synchronization error; Sec. 4.5 corrected to +0.114+0.114 / +0.111+0.111
Structural inputs unspecifiedPer-tool inputs now stated (Sec. 3.3, App. A.14)
External pretraining overlapAudited and disclosed (App. A.14)
Unequal valid-target countsPaired comparison on common targets added (Sec. 4.2)
Reward attribution undefinedFormulas and weights written out (Sec. 3.8, App. A.15)
Limited pocket-level gainsRe-evaluated with a train-derived pocket size (Table 3)

W1. The independent contribution of LLM orchestration

We agree, and we have changed the claim rather than defended it. Details in Q1 below; the short version is that on matched targets the LLM selector is statistically indistinguishable from a fixed seven-tool core and from a learned non-LLM selector. The abstract, Sec. 2, Sec. 4.4 and the conclusion now describe learned fusion as the source of the gain and LLM selection as a usable interface, not an accuracy mechanism.

W2. Consistency and evaluation validity

All four items are addressed below (Q2–Q5). The headline itself was re-verified first: using the saved Maestro v4 selections and the frozen LightGBM checkpoint (fullsystem_predictions_v2/model.txt), we reproduced all 107/107 per-target correlations, mean 0.593082242990.59308224299. App. A.13 now records which tables come from that checkpoint and which are separate runs.


Q1. Independent value of LLM-based selection

Three alternatives on the same 107 held-out targets, with 10,000 PDB-grouped paired bootstrap resamples (Table 29):

Selection methodPearson rrΔ\Delta vs. Maestro (95% CI)
Maestro (LLM)0.59308—
Fixed seven-tool core0.59407+0.00099 [−0.00501, 0.00625]+0.00099\ [-0.00501,\ 0.00625]
Non-LLM KNN selector0.59031−0.00277 [−0.01166, 0.00476]-0.00277\ [-0.01166,\ 0.00476]
Random extra tools (30 runs)0.58823−0.00485 [−0.01083, 0.00069]-0.00485\ [-0.01083,\ 0.00069]

KNN and random use about nine tools per target against 8.948.94 for Maestro, and the core uses seven, so these are comparable tool counts rather than strictly equalized runtimes; the caption says so.

Target-level behaviour. On 8inf_u_x, Maestro adds GraphBind (individual r=0.6190r = 0.6190) and the fused prediction rises from 0.30680.3068 with the core alone to 0.48000.4800. On 8cdr_NN_Bb the extra tools hurt, 0.0934→−0.00140.0934 \rightarrow -0.0014. Aggregated, the selected set contains the retrospectively best single tool for 92/10792/107 targets (86.0%86.0\%) against 86/10786/107 (80.4%80.4\%) for the core; of the 21 targets whose best tool lies outside the core, Maestro recovers six.

Predictability of the best tool. We should be precise about what this measures: it is best-tool coverage by a set of roughly nine tools, not identification of the best tool. It does not establish that the best predictor is identifiable from target characteristics, so we removed that claim from the abstract rather than leaving it unquantified.

(continued in part 2)

AuthorsOpenReview ↗

Response to Reviewer Mqbn (2/3)

(continued from part 1)

Q2. Consistency of experimental results

Tables 15–18 vs. Table 2. The sub-group tables were produced under an earlier configuration with an additional polishing step (overall r=0.587584r = 0.587584); Table 2 reports the formal configuration (r=0.593082r = 0.593082). Regenerated from the formal predictions with group memberships and counts unchanged:

TableCorrected RiboSeer rr (group size)
15: RNA lengthShort 0.5956 (28); Medium 0.7220 (50); Long 0.3684 (29)
16: Protein domainRRM 0.4934 (8); KH −0.0271-0.0271 (1); ZnF 0.5645 (4); Other 0.6094 (94)
17: RNA contextStem-loop 0.6041 (83); Unstructured 0.5548 (24)
18: DifficultyEasy 0.5235 (7); Medium 0.3713 (16); Hard 0.6411 (84)

The sample-weighted mean of each is now ≈0.5931\approx 0.5931, matching Table 2.

Sec. 4.5 vs. Table 20. A synchronization error: Table 20 was updated during revision and the running text was not. The correct margins are +0.114+0.114 (temporal, 0.4310.431 vs. 0.3170.317) and +0.111+0.111 (tight 25% identity, 0.4070.407 vs. 0.2960.296). Sec. 4.5 now matches. These alternative splits retrain the fusion per partition, so unlike the main evaluation they do not come from the default-split frozen checkpoint, which App. A.13 now states.

Q3. Independence and fairness

Structural inputs. Now explicit in Sec. 3.3 and App. A.14. P2Rank, Fpocket, DeepPocket, GraphBind, EquiPNAS and NucleicNet receive experimental protein coordinates extracted from the native complex; HDOCK and HADDOCK 3 receive protein and RNA coordinates from the same complex, extracted separately. These are bound-state coordinates, not unbound or independently predicted ones. Boltz-2, Chai-1 and the other Category A tools receive sequences only. We no longer describe the benchmark as sequence-only.

External training-data overlap. 3wbm_A_X and 4n0t_A_B appear in GraphBind's published RNA-495 training set at the PDB and chain level, and EquiPNAS reports RNA-495 as a training source, though we could not verify the exact evaluated checkpoint's composition. 2czj_E_F is sequence-identical to an RNABindRPlus RB198 entry. Several tools publish no checkpoint-specific training list, so overlap cannot be excluded for them. Independence of our own split does not imply independence from every external predictor's pretraining data, and App. A.14 now says this plainly.

Paired comparison on common targets. On the 104 targets where Boltz-2 is valid, RiboSeer reaches 0.594160.59416 against 0.438980.43898, a paired gain of +0.15518+0.15518 (95% PDB-grouped bootstrap CI [0.12127,0.18778][0.12127, 0.18778], 10,000 resamples). This is now in Sec. 4.2 and in the abstract. It removes the unequal-nn concern for the headline comparison, though not the structural-input or external-overlap caveats above.

(continued in part 3)

AuthorsOpenReview ↗

Response to Reviewer Mqbn (3/3)

(continued from part 2)

Q4. Validity of adaptive feedback

Attribution. Verdict combines five label-free scores with weights (0.25,0.20,0.15,0.25,0.15)(0.25, 0.20, 0.15, 0.25, 0.15), renormalized over available components. The structural score is attributed in full to Category A tools and at 50%50\% to others; physicochemical, conservation and motif use the Step-5 tool weights with equal allocation as fallback; consensus comes from each tool's mean pairwise Jaccard. Memory then applies, per successful tool kk, component mm, category jj,

Wk,m,j(t+1)=(1−η)Wk,m,j(t)+η sk,m,j(t),η=0.1,W_{k,m,j}^{(t+1)}=(1-\eta)W_{k,m,j}^{(t)}+\eta\, s_{k,m,j}^{(t)},\qquad \eta=0.1,

leaving failed tools and missing components unchanged (Sec. 3.8, App. A.15).

Does the reward track quality? Partly. Over 106 targets with usable saved records, the aggregate reward correlates with fused quality at Spearman ρ=0.210\rho = 0.210 ([0.026,0.407][0.026, 0.407]), and the consensus component at 0.3390.339 ([0.139,0.525][0.139, 0.525]). Across 1,515 tool–target observations, tool-specific consensus correlates with tool accuracy at 0.1440.144 ([0.098,0.188][0.098, 0.188]), rising to 0.6390.639 for Boltz-2 and 0.6440.644 for Chai-1. Positive but weak, from a historical run rather than the frozen evaluation.

Table 14. Your doubt about the modest Memory gain was well placed, and the reason is worse than modest effect size. The streaming code computes tool rewards from Pearson correlation with each test target's ground-truth labels, not from the label-free Verdict reward, and it also uses a non-LLM UCB selector and a separately trained fusion; the original launch command is unavailable. The 0.527→0.5350.527 \rightarrow 0.535 difference therefore cannot be read as the effect of label-free online adaptation, and it is not comparable with the headline, which uses frozen selections and a frozen fusion with no online updates. Table 14 now carries this correction and we make no claim of demonstrated label-free online learning.

Q5. Binding pocket localization

Two reasons for the gap you noticed. The fusion is trained for residue scores, with no geometric objective; and pocket extraction groups high-scoring residues by sequence proximity, not 3D proximity, so better residue ranking need not translate proportionally into better pocket centres.

More importantly, the submitted procedure chose kk equal to each test target's true number of binding residues, which is not usable on an unseen target. We re-ran the evaluation with kk fixed from the 225 training targets only (7.762%7.762\% of protein length); test labels enter only when scoring the extracted pocket (Table 3):

MethodValidDCC (Å) ↓DCA (Å) ↓IoU ↑Top-1 ↑Top-3 ↑
Boltz-210216.678.750.06240.98%6.86%
RiboSeer10513.517.990.06915.71%11.43%

On the 102 jointly valid targets: DCC 13.4913.49 vs. 16.6716.67, IoU 0.06670.0667 vs. 0.06240.0624, Top-1 5.88%5.88\% vs. 0.98%0.98\%, Top-3 11.76%11.76\% vs. 6.86%6.86\%. So without any test-derived pocket size the ordering is now consistent with the residue-level result, while absolute success stays low for both methods, which we state rather than smooth over. The label-sized protocol reproduced the historical Boltz-2 values but not exactly the historical RiboSeer ones, so we present the new evaluation as a separate corrected result and retain the superseded table as Table 30.


The revision keeps the claims that survived the audit and drops those that did not: the benchmark and the fusion result, now reproducible from a hashed checkpoint and supported by paired CIs on common targets, are presented as the contribution, and agentic orchestration is presented as an interface with measured, limited effect. We are happy to run further checks during the discussion period.

Thank you again for helping us identify these issues. If the corrected results and additional validation resolve your main concerns, we hope you will consider revising your score upward.

Best regards,

Authors

Thank you for the thorough and transparent response; I particularly appreciate the systematic integration and evaluation of 15 heterogeneous predictors, which provides valuable empirical evidence for cross-paradigm fusion. However, given the limited independent methodological contribution and remaining evaluation concerns, I will maintain my rating of 2.

Reviewer_zEX6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Proposed RiboSeer system combines 15 protein-RNA interface predictors through LLM-based tool selection, a mandatory seven-tool core, supervised residue-level fusion, and reliability updates. On 107 test targets, it reports PCC r=0.593 vs 0.439 for the best individual tool and 0.515 for a weighted ensemble. The main contribution is an empirical integration and benchmarking study.

Critical strengths
  • Evaluating four complementary predictor families under common contact labels is practically useful.
  • The ablations identify supervised fusion as the main source of improvement and expose limited gains in pocket geometry.
Critical weaknesses
  • The distinct scientific contribution is not established. The method combines existing predictors, supervised stacking, UCB selection, and EMA updates. Input-dependent ensemble selection already exists; META-DES selects competent classifiers. Within this paper, per-residue features alone reach 0.589, and logistic regression on engineered features reaches 0.587. Table 11 includes an all-tools fusion baseline but fixed-core fusion and a competitive learned selector without an LLM are missing. The reported 0.039 win over UCB, therefore, does not establish a benefit specific to language-model reasoning.

  • The generalization and adaptation protocols are insufficiently specified. Spectral partitioning of a TM-score affinity graph does not itself guarantee that every train-test pair falls below the similarity threshold. Overlap with the pre-trained predictors training data and template databases is not audited. For online adaptation, the conversion of the 5 reward components into tool-specific reliability is undefined, and the reward relationship to actual prediction quality is untested.

  • Several empirical claims require reconsideration. Weighting the sub-group means in Tables 15-18 by their sample counts gives approximately 0.587-0.588, rather than the headline 0.593; rounding cannot explain this. Pocket IoU and top-1 success do not improve over the best baseline. Furthermore, MCC uses a per-target optimal threshold, and probability calibration is claimed without calibration measurements.

Questions for authors
  • Does the LLM outperform fixed seven-tool fusion and a learned non-LLM selector under matched computational budgets? Kindly provide paired confidence intervals across targets.

  • What is the maximum train-test TM-score? Which native coordinates and meta-data reach the predictors, prompts and reward computation?

  • Please specify reward formulas, weights, tool-specific attribution, initialization, resets, and sample ordering. Why does default streaming performance reach 0.535 while the headline reaches 0.593, given that held-out inference should already freeze fusion parameters?

  • Can all tables be regenerated from one documented checkpoint? Please also report performance with validation-selected thresholds and assess calibration.

5 comments
AuthorsOpenReview ↗

Response to Reviewer zEX6 (1/4)

Dear Reviewer zEX6,

Thank you for a review that was specific enough to be actionable. Your three concerns sent us back to the code and the saved artifacts rather than to the text, and the audit found real errors. We have uploaded a revised PDF in which every change is marked in blue. The summary is:

Issue you raisedOutcome
Headline vs. sub-group meansTraced to a stale configuration; sub-group tables regenerated, weighted mean now 0.59310.5931 (App. A.13, Tables 15–18)
Claim of LLM-specific benefitNew controlled selection study; claim withdrawn (Sec. 4.4, Table 29)
Missing fixed-core and learned non-LLM selectorsBoth added, with paired CIs (Table 29)
Strict TM-score separationClaim corrected with measured values (Sec. 4.1)
Overlap with predictors' training dataAudited and reported (App. A.14)
Undefined reward attributionFormulas and weights now specified (Sec. 3.8, App. A.15)
Pocket size from test labelsRe-evaluated with a train-derived pocket size (Table 3)
"Calibrated probability"Terminology corrected throughout; no calibration claimed
Test-optimal MCC thresholdReplaced by a train-selected threshold (Table 22 caption)
One documented checkpointCheckpoint hashed; 107/107 targets reproduced (App. A.13)

The one thing the audit did not overturn is the main empirical result. It reproduces exactly from a single frozen checkpoint, it survives the paired comparison you asked for, and it survives the corrected pocket protocol. What it buys is a smaller claim than the submission made, and we have rewritten the paper to make that claim.


W1. What the distinct contribution is

We accept that stacking, UCB, EMA and LLM routing are off-the-shelf. The contribution we can defend is the integration and comparative evaluation itself: 15 predictors from four paradigms, with incompatible output types and failure modes, put under one residue-level label definition with adapters, a fallback for failed tools, and a learned fusion over the standardized outputs. Related work now situates this against dynamic ensemble selection as the established non-LLM form of input-dependent selection, and identifies our KNN selector as a simple instance of it (Sec. 2). We describe it there without claiming a benchmark against any particular DES method, since we did not run one.

On fusion, we re-ran all four aggregators on matched inputs (App. A.13, Table 28):

FusionPearson rrLightGBM gain, paired 95% CI
Simple average0.4741[0.0824, 0.1542][0.0824,\ 0.1542]
Train-weighted average0.5162[0.0482, 0.1041][0.0482,\ 0.1041]
Ridge regression0.5788[−0.0004, 0.0289][-0.0004,\ 0.0289]
LightGBM0.5931—

The reading we now take, and state in Sec. 4.2, is that learning the fusion is what matters, not which learner. That is consistent with your observation that logistic regression reaches 0.5870.587 and per-residue features alone 0.5890.589: the engineered feature space plus a fitted combiner carries the result, and we no longer present the learner choice as a contribution.

W2. Generalization and adaptation

Similarity. You are right that spectral partitioning of a thresholded affinity graph does not certify a pairwise bound. Measured on our split, the maximum cross-split TM-score is 0.8230.823 and 2,9782{,}978 of 24,07524{,}075 train–test pairs reach 0.350.35. A test target's maximum similarity to training does not predict its accuracy (Spearman ρ=0.037\rho = 0.037, p=0.70p = 0.70), which we report as weak evidence against a similarity effect rather than as a clearance. Sec. 4.1 now states this instead of claiming strict disjointness, and the abstract, introduction and conclusion no longer use "structurally disjoint".

External overlap. Audited and reported in App. A.14: 3wbm_A_X and 4n0t_A_B appear in GraphBind's published RNA-495 training set at the PDB and chain level, and EquiPNAS lists RNA-495 as a source; 2czj_E_F is sequence-identical to an RNABindRPlus RB198 entry. Several tools publish no checkpoint-specific training list, so for those we cannot exclude overlap and say so. This is a limitation of benchmarking published checkpoints, and the honest position is to disclose it rather than to assert independence we cannot verify.

Adaptation. Both of your points here were correct. The reward decomposition is now written out (below and in Sec. 3.8), and its relationship to prediction quality is now measured rather than assumed: over 106 targets with usable saved records, the aggregate reward correlates with fused quality at Spearman 0.2100.210 (95% CI [0.026,0.407][0.026, 0.407]), with the consensus component the strongest contributor at 0.3390.339 ([0.139,0.525][0.139, 0.525]) (App. A.15). That is a weak positive signal, not a validated objective, and the paper now says so.

(continued in part 2)

AuthorsOpenReview ↗

Response to Reviewer zEX6 (2/4)

(continued from part 1)

W3. Consistency, pocket accuracy, calibration

The 0.5930.593 / 0.5880.588 gap. Not rounding, as you suspected. The sub-group tables were generated under an earlier configuration with an additional polishing step (overall r=0.587584r = 0.587584); Table 2 is the formal configuration without it (r=0.593082r = 0.593082). We regenerated Tables 15–18 from the formal predictions, keeping group memberships and sample counts fixed, and the sample-weighted mean of each is now 0.59310.5931. The provenance of every table is documented in App. A.13.

Pocket localization. The more serious finding is the one you anticipated: the submitted pocket extraction set kk to each test target's true number of binding residues. We replaced it with kk derived from training data alone (7.762%7.762\% of protein length, the binding-residue fraction over the 225 training targets); test labels now enter only when scoring the extracted pocket. Table 3:

MethodValidDCC (Å) ↓DCA (Å) ↓IoU ↑Top-1 ↑Top-3 ↑
Boltz-210216.678.750.06240.98%6.86%
RiboSeer10513.517.990.06915.71%11.43%

On the 102 jointly valid targets the ordering is the same (DCC 13.4913.49 vs. 16.6716.67; Top-1 5.88%5.88\% vs. 0.98%0.98\%). So the corrected protocol is in fact better for us than the submitted one, but absolute success stays low, and Sec. 4.3 explains why: the fusion is trained for residue scores, and segments are grouped in sequence rather than in 3D. The superseded table is retained as Table 30 so the change is auditable.

Thresholds and calibration. The per-target optimal threshold is not deployable and we have stopped reporting it as a result. Selecting one threshold by five-fold out-of-fold F1 on training gives 0.16720.1672; applied unchanged to test, MCC is 0.53660.5366 (against 0.3840.384 at the default 0.50.5). On calibration: the fusion is LGBMRegressor.predict with no sigmoid and no post-hoc calibration, and a few scores fall outside [0,1][0,1]. Eq. 15 and the surrounding text now describe a squared-error regressor producing continuous binding scores; "calibrated probability" is removed from the abstract, introduction, Sec. 3.7 and Fig. 1. We do not report ECE or Brier, since computing them on unbounded regression scores would not be meaningful.


(continued in part 3)

AuthorsOpenReview ↗

Response to Reviewer zEX6 (3/4)

(continued from part 2)

Q1. LLM vs. fixed core and a learned non-LLM selector

Run on the same 107 targets, with 10,000 PDB-grouped paired bootstrap resamples (Table 29):

Selection policyPearson rrΔ\Delta vs. Maestro (95% CI)
Maestro (LLM)0.59308—
Fixed seven-tool core0.59407+0.00099 [−0.00501, 0.00625]+0.00099\ [-0.00501,\ 0.00625]
Non-LLM KNN selector (train features)0.59031−0.00277 [−0.01166, 0.00476]-0.00277\ [-0.01166,\ 0.00476]
Random extra tools (30 runs)0.58823−0.00485 [−0.01083, 0.00069]-0.00485\ [-0.01083,\ 0.00069]

No interval excludes zero. Budgets are comparable but not strictly matched: KNN and random use about nine tools per target against 8.948.94 for Maestro, the core uses seven, and the core row reuses the frozen fusion while KNN and random required separately trained fusions. We report these caveats in the caption rather than claim matched compute.

Our conclusion is the one your review points to. The +0.039+0.039 over the UCB-only ablation does not isolate language-model reasoning, because the mandatory core is doing the work in both arms. The abstract, introduction, Sec. 2, Sec. 4.4 and the conclusion now state that LLM selection is statistically indistinguishable from a fixed core and from a non-LLM selector, and present it as a usable interface rather than a source of accuracy. What the selector does provide is interpretable per-target coverage: its set contains the retrospectively best single tool for 86.0%86.0\% of targets against 80.4%80.4\% for the core, and the per-target effect cuts both ways (on 8inf_u_x adding GraphBind lifts fused rr from 0.3070.307 to 0.4800.480; on 8cdr_NN_Bb the extras drop it from 0.0930.093 to −0.001-0.001).

Q2. Maximum TM-score and what reaches the predictors

Maximum cross-split TM-score 0.8230.823; 2,978/24,0752{,}978/24{,}075 pairs at or above 0.350.35; similarity-accuracy Spearman ρ=0.037\rho = 0.037, p=0.70p = 0.70.

Inputs, now stated explicitly in Sec. 3.3 and App. A.14: P2Rank, Fpocket, DeepPocket, GraphBind, EquiPNAS and NucleicNet receive experimental bound-state protein coordinates from the native complex; HDOCK and HADDOCK 3 receive protein and RNA coordinates from the same complex, extracted separately; Boltz-2, Chai-1 and the other Category A tools receive sequences only and fold their own structures. The benchmark is therefore not sequence-only, and we no longer describe it as such.

For the agent: the inspected Scope prompt contains sequences and lengths, with no atomic coordinates and no test binding labels, and the saved Maestro v4 categories agree with Scope-derived categories on all 107 targets. A separate preprocessing script does contain test-label-derived counts, but the saved selections match Scope, not that script. Complete historical prompts and the provenance of the initial UCB weight tensor were not recovered, so we state in App. A.14 that indirect leakage through those initial weights cannot be fully excluded. We would rather flag this than assert an independence we cannot demonstrate. It does not affect the formal evaluation, which uses frozen selections and a frozen fusion.

(continued in part 4)

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers and AC (1/2)

Dear Reviewers and Area Chair,

Thank you for three reviews that converged on the same questions. We treated them as an audit request rather than a presentation problem: we went back to the saved predictions, the model checkpoint, the tool adapters, the prompts and the evaluation scripts, and checked each claim against what the code actually does. The audit found real errors, including two we would not have caught otherwise. All of them are now corrected in the revised PDF, with every change marked in blue.

This part gives the change log and the first two shared concerns; part 2 covers the remaining three and summarizes what we now claim. Per-reviewer responses are posted separately.

What changed in the revision

ChangeWhere
New controlled tool-selection study (core-only, random extras, non-LLM KNN) with paired CIsSec. 4.4, Table 29
Claims about LLM orchestration rewrittenAbstract, Sec. 1, 2, 4.4, 5
Sub-group tables regenerated from the formal checkpointTables 15–18, App. A.13
Sec. 4.5 split margins corrected to +0.114+0.114 / +0.111+0.111Sec. 4.5
TM-score separation claim corrected with measured valuesSec. 4.1
Overlap with external predictors' training data auditedApp. A.14
Per-tool structural inputs stated (bound-state vs. sequence)Sec. 3.3, App. A.14
Paired comparison on commonly valid targetsSec. 4.2, abstract
Reward formulas, weights and tool attribution written outSec. 3.8, App. A.15
Pocket evaluation redone with a train-derived pocket sizeTable 3, Table 30 (superseded)
"Calibrated probability" replaced by continuous binding scoreEq. 15, Sec. 3.7, Fig. 1
Train-selected MCC threshold instead of per-target optimalTable 22
Checkpoint hashed, provenance of every table documentedApp. A.13
Enlarged Figure 1 added; simplified redraw plannedApp. A.17

Common concern 1: the contribution of LLM-based orchestration All three reviews questioned whether the 0.554→0.5930.554 \rightarrow 0.593 ablation isolates language-model reasoning. It does not, because the mandatory seven-tool core is present in both arms. We therefore ran the comparison directly on the same 107 test targets, with 10,000 PDB-grouped paired bootstrap resamples (Table 29):

Selection policyPearson rrΔ\Delta vs. Maestro (95% CI)
Maestro (LLM)0.59308—
Fixed seven-tool core0.59407+0.00099 [−0.00501, 0.00625]+0.00099\ [-0.00501,\ 0.00625]
Non-LLM KNN selector0.59031−0.00277 [−0.01166, 0.00476]-0.00277\ [-0.01166,\ 0.00476]
Random extra tools (30 runs)0.58823−0.00485 [−0.01083, 0.00069]-0.00485\ [-0.01083,\ 0.00069]

No interval excludes zero. We have rewritten the abstract, introduction, related work, Sec. 4.4 and the conclusion accordingly: learned fusion over a paradigm-diverse pool supplies the gain, and LLM selection is a usable interface, not a source of accuracy. We also no longer claim that the best tool is predictable from target characteristics. What we do report, as behaviour rather than benefit, is that the selected set contains the retrospectively best single tool for 86.0%86.0\% of targets against 80.4%80.4\% for the core alone, and that the per-target effect runs both ways (8inf_u_x: 0.307→0.4800.307 \rightarrow 0.480; 8cdr_NN_Bb: 0.093→−0.0010.093 \rightarrow -0.001).

On the fusion side, we re-ran all aggregators on matched inputs (Table 28): simple average 0.4740.474, train-weighted average 0.5160.516, Ridge 0.5790.579, LightGBM 0.5930.593. LightGBM's gain is supported over the two averages (paired CIs [0.082,0.154][0.082, 0.154] and [0.048,0.104][0.048, 0.104]) but not over Ridge ([−0.000,0.029][-0.000, 0.029]). The honest reading, now in Sec. 4.2, is that learning the fusion matters and the choice of learner does not.

Common concern 2: reconciling the reported numbers Both discrepancies were real.

Headline vs. sub-group tables. Tables 15–18 came from an earlier configuration with an additional polishing step (overall r=0.587584r = 0.587584), while Table 2 reports the formal configuration (r=0.593082r = 0.593082). We regenerated the sub-group tables from the formal predictions with group memberships and counts unchanged; each now has a sample-weighted mean of ≈0.5931\approx 0.5931.

Sec. 4.5 vs. Table 20. A manuscript synchronization error. The correct margins are +0.114+0.114 (temporal, 0.4310.431 vs. 0.3170.317) and +0.111+0.111 (tight 25% identity, 0.4070.407 vs. 0.2960.296). These splits retrain the fusion per partition, which App. A.13 now states.

Provenance. The headline reproduces exactly from saved Maestro v4 selections and one frozen LightGBM checkpoint (fullsystem_predictions_v2/model.txt, SHA256 063d497e…cebe79): all 107/107 per-target correlations, mean 0.593082242990.59308224299. App. A.13 lists which tables come from this checkpoint and which are separate runs, so no table is presented as something it is not.

(continued in part 2)

AuthorsOpenReview ↗

General Response to All Reviewers and AC (2/2)

(continued from part 1)

Common concern 3: independence and fairness of the evaluation Within-benchmark similarity. Spectral partitioning of a thresholded TM-score graph does not certify a pairwise bound. Measured on our split: maximum cross-split TM-score 0.8230.823, and 2,9782{,}978 of 24,07524{,}075 train–test pairs at or above 0.350.35. A test target's maximum similarity to training does not predict its accuracy (Spearman ρ=0.037\rho = 0.037, p=0.70p = 0.70). Sec. 4.1 now reports these numbers, and "structurally disjoint" is removed from the abstract, introduction and conclusion.

Inputs to each predictor. Now stated per tool. P2Rank, Fpocket, DeepPocket, GraphBind, EquiPNAS and NucleicNet receive experimental bound-state protein coordinates from the native complex; HDOCK and HADDOCK 3 receive protein and RNA coordinates from the same complex, extracted separately; Boltz-2, Chai-1 and the other Category A tools receive sequences only and fold their own structures. The benchmark is not sequence-only and we no longer describe it that way.

External pretraining overlap. 3wbm_A_X and 4n0t_A_B appear in GraphBind's published RNA-495 training set at the PDB and chain level, and EquiPNAS lists RNA-495 as a source; 2czj_E_F is sequence-identical to an RNABindRPlus RB198 entry. Several tools publish no checkpoint-specific training list, so overlap cannot be excluded for them. Independence of our split does not imply independence from every external checkpoint, and App. A.14 says so.

Unequal valid-target counts. On the 104 targets where Boltz-2 is valid, RiboSeer reaches 0.5940.594 against 0.4390.439, a paired gain of +0.155+0.155 (95% CI [0.121,0.188][0.121, 0.188]). This is now in Sec. 4.2 and the abstract, so the headline comparison uses a common set.

Common concern 4: pocket localization Two reviewers noted that pocket-level gains did not match residue-level gains. The audit found a worse problem: the submitted procedure set the number of selected residues to each test target's true binding-residue count. We replaced it with a size fixed from the 225 training targets only (7.762%7.762\% of protein length); test labels now enter only when scoring the extracted pocket (Table 3):

MethodValidDCC (Å) ↓DCA (Å) ↓IoU ↑Top-1 ↑Top-3 ↑
Boltz-210216.678.750.06240.98%6.86%
RiboSeer10513.517.990.06915.71%11.43%

On the 102 jointly valid targets the ordering holds (DCC 13.4913.49 vs. 16.6716.67; Top-1 5.88%5.88\% vs. 0.98%0.98\%). With no test-derived quantity, RiboSeer now leads on every pocket metric, while absolute success stays low for both. Sec. 4.3 explains why: no geometric objective in the fusion, and segments grouped in sequence rather than in 3D. The superseded table is Table 30.

Common concern 5: calibration, thresholds, and adaptive feedback Calibration. The fusion is LGBMRegressor.predict with no sigmoid and no post-hoc calibration, and a few scores fall outside [0,1][0,1]. Eq. 15 and the surrounding text now describe a squared-error regressor producing continuous binding scores; "calibrated probability" is removed, and we do not report ECE or Brier on unbounded scores.

Thresholds. The per-target optimal MCC threshold used test labels and is not deployable. A threshold chosen by five-fold out-of-fold F1 on training (0.16720.1672), applied unchanged to test, gives MCC 0.53660.5366 (against 0.3840.384 at 0.50.5).

Online adaptation. The reward is now fully specified: five components with weights (0.25,0.20,0.15,0.25,0.15)(0.25, 0.20, 0.15, 0.25, 0.15), renormalized over available components, with tool-specific attribution and the EMA update at η=0.1\eta = 0.1 (Sec. 3.8, App. A.15). Its validity is measured rather than assumed: Spearman 0.2100.210 ([0.026,0.407][0.026, 0.407]) against fused quality, consensus strongest at 0.3390.339. The streaming table (Table 14) turned out to compute tool rewards from test ground-truth labels, not from the label-free reward, so it does not measure label-free online adaptation; the table now carries that correction and the claim of demonstrated online gains is withdrawn. The headline freezes selections and the fusion, with no online updates.

What we now claim A single-protocol benchmark of 15 predictors across four paradigms, and a learned fusion that reaches r=0.593r = 0.593 and AUROC 0.8990.899 on 107 held-out targets, reproduces exactly from one hashed checkpoint, gains +0.155+0.155 ([0.121,0.188][0.121, 0.188]) over the best single tool on commonly valid targets, holds across four splits, four cutoffs and four strata, and leads on every pocket metric once pocket size comes from training data. Orchestration is reported as an interface with no measurable accuracy benefit over a fixed core, and the limits on similarity, external overlap and online adaptation are stated in the paper rather than left implicit.

We are glad to run further experiments during the discussion period.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.