- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents a framework for RNA 3D structure prediction that combines RNA-adapted Protenix, multi-conformation supervision, synthetic structural augmentation, and Agentic Tree Search (ATS) for selecting structural predictions.
The authors fine-tune Protenix using curated RNA structures, MSA-assisted training, and a multi-conformation objective that selects the best-matching experimental reference. At inference time, Protenix, template-based modeling (TBM), and Boltz each generate five candidate structures. ATS then selects five predictions from this pool using source-model reliability weights, structural diversity, and fixed structural priors.
The proposed framework achieves public/private TM-scores of 0.684/0.632 on the Stanford RNA 3D Folding benchmark, compared with 0.601/0.591 for the reported Kabsch-based selection baseline. The paper also provides ablations on multi-conformation supervision, MSA-assisted training, and ATS parameter sensitivity.
Critical strengths
1. Important and practically relevant problem.
RNA 3D structure prediction remains challenging due to limited experimental structural data, conformational flexibility, and complex tertiary interactions. The paper addresses these difficulties in a relevant benchmark setting where multiple structural predictions are required.
2. Promising empirical improvements from combining complementary predictors.
The integration of learned predictors and template-based modeling is well motivated. The reported improvement from 0.601/0.591 to 0.684/0.632 when replacing the simpler selection strategy with ATS is substantial and suggests that candidate selection can meaningfully affect prediction accuracy.
3. Useful ablation studies for multi-conformation supervision and MSA-assisted training.
Appendix N.1 provides matched ablations showing that both multi-conformation supervision and MSA-assisted training contribute to prediction accuracy. In particular, multi-conformation supervision improves the private TM-score from 0.381 to 0.412, and combining it with MSA-assisted training further improves the score to 0.424. These experiments provide useful evidence for the effectiveness of the proposed training choices.
Critical weaknesses
W1. The methodological novelty and independent contribution of Agentic Tree Search are insufficiently established.
ATS is presented as a central methodological contribution. However, Algorithm 1 describes a greedy, diversity-aware candidate selection procedure rather than a conventional tree search algorithm. The method initializes the selected set with fixed structural anchors and iteratively adds the candidate maximizing a weighted combination of structural diversity and distance from these anchors. The authors explicitly acknowledge that the current implementation does not maintain multiple search branches. This procedure appears closely related to existing greedy quality-diversity selection methods, including Maximal Marginal Relevance (MMR)-style re-ranking. While incorporating structural priors and source-model reliability weights may be practically useful, it remains unclear what fundamentally distinguishes ATS from these established approaches.
More importantly, the current experiments do not adequately isolate the contribution of the proposed selection procedure from its heuristic components. The comparison with Kabsch-based top-confidence selection is informative, but it does not establish that ATS outperforms stronger diversity-aware selection baselines under matched conditions. For example, Appendix N.2 demonstrates substantial sensitivity to source-model weights and anchor choices. Since the reference configuration initializes the selected set with Protenix and TBM, two of the final five predictions are effectively predetermined. It is therefore important to disentangle improvements arising from anchor selection, reliability weighting, and diversity-aware greedy selection.
The absence of a TBM-only baseline also makes the contribution of the individual components difficult to assess, particularly given the substantial improvement observed after introducing TBM into the ensemble.
I believe the paper would be considerably strengthened by:
- Comparing ATS against standard diversity-aware selection methods, such as MMR-style re-ranking, farthest-point selection, and clustering-based representative selection, using the same 15-candidate pool.
- Reporting TBM-only performance and component-wise ablations of the ATS scoring function.
- Providing an oracle best-of-15 analysis to quantify the gap between candidate quality and the effectiveness of the selection strategy.
These experiments are important for determining whether ATS represents a distinct methodological contribution or primarily an effective combination of existing selection heuristics.
W2. The experimental evidence does not fully establish the claimed structural generalization.
The title and conclusions emphasize generalizable RNA 3D structure prediction. However, the experiments do not sufficiently distinguish generalization to unseen RNA sequences from generalization to evolutionarily distant sequences or previously unseen structural folds. In Section 4.1 and Appendix A.2, the authors explicitly acknowledge that they do not apply a global sequence-identity decontamination threshold between training and evaluation sequences. Although known evaluation structures are excluded using their PDB identifiers, closely related sequences and structural homologs may remain in the training corpus. This concern is particularly relevant because TBM makes an important contribution to the final framework. Template-based prediction is a legitimate and useful approach, but strong performance on targets with closely related structural templates does not necessarily demonstrate comparable performance on RNA families or folds lacking suitable templates.
The external evaluations on CASP15, RNA-Puzzles, and YangLab30 provide additional evidence, but only aggregate TM-scores are reported, without detailed comparisons against strong reference methods under matched conditions. Similarly, the post-training PDB analysis demonstrates complementary behavior between TBM and Protenix but does not directly evaluate the complete ATS framework. Therefore, I would like to see performance stratified by sequence similarity to the training corpus, structural template availability, and RNA length. In particular, evaluating targets with low sequence similarity and limited template support would help clarify the extent to which the proposed framework generalizes beyond previously observed structural families.
W3. Important aspects of the evaluation protocol and experimental controls require clarification.
The paper reports private benchmark results for numerous configurations, including ATS weight and anchor sensitivity experiments. However, the provenance of these private evaluations and whether private feedback influenced model or hyperparameter selection are not sufficiently clear. Without a precise description of which test set and submission procedure were used, it is difficult to assess the independence of the reported comparisons.
There is also ambiguity surrounding the MSA-free inference setting. The authors attribute the absence of inference-time MSA to the lack of internet access in the Kaggle environment. However, the competition's official dataset description states that precomputed MSA files are made available for hidden test sequences. The inability to perform online MSA generation therefore does not, by itself, establish that inference-time MSA was unavailable.
These issues do not invalidate the reported results, but clarifying the evaluation protocol and the rationale for MSA-free inference is important for interpreting the findings and ensuring reproducibility.
Questions for authors
Q1. Private benchmark evaluation and hyperparameter selection. Could the authors clarify how the reported private benchmark scores were obtained? Specifically, were they produced through official final Future Data evaluation, post-competition late submissions, or another evaluation procedure? Were all configurations evaluated on the same private target set? Additionally, were private benchmark results used to select ATS source-model weights, structural anchors, or other hyperparameters? Clarifying the tuning procedure and evaluation chronology would strengthen confidence in the reported results.
Q2. MSA availability and inference-time evaluation. The competition's official data description indicates that precomputed MSAs are available for hidden test sequences, even though online sequence database searches are unavailable. Could the authors explain why these precomputed MSAs were not used during inference? Was this a deliberate methodological choice, a model compatibility issue, or a computational constraint? If feasible, comparing inference with and without the provided MSAs would help clarify the significance of MSA-assisted training followed by MSA-free deployment.
Q3. Multi-conformation supervision and the definition of distinct conformational states. The proposed multi-conformation objective minimizes the structural loss over available references, with an auxiliary loss on the first reference. While this allows predictions to match alternative experimental states, it does not explicitly encourage different generated predictions to cover distinct conformations. Could the authors evaluate whether multi-conformation supervision actually improves conformational diversity or coverage, rather than only best-of-five prediction accuracy? Relatedly, Appendix A reports that 3,009 of 3,203 filtered unique sequences exhibit multiple experimentally observed conformations. This proportion appears unusually high and would benefit from further explanation. How does the criterion based on structural variation distinguish biologically meaningful alternative conformations from minor experimental or geometric differences? Reporting the distribution of pairwise TM-scores among reference conformations would help clarify the nature of the multi-conformation training data.
Q4. Synthetic structural augmentation and controlled comparisons. The paper reports generating 3,982 synthetic structural candidates and retaining 400 for training. However, the generation procedure and quality filtering criteria are not described in enough detail for independent reproduction. Could the authors provide the generation architecture, training procedure, structural filtering criteria, and selection thresholds? Moreover, the transition from Merged-v1 to Merged-v2 simultaneously introduces synthetic augmentation and removes approximately 40 CASP16-derived sequences. A controlled experiment changing only the synthetic augmentation component would be necessary to isolate its contribution. Such an experiment could also help explain why synthetic augmentation improves the public score but not the corresponding private score.
Q5. DRfold and ResidueGNN evaluation. The paper motivates ResidueGNN by stating that DRfold does not directly provide the C1′ coordinates required by the benchmark. However, the original DRfold framework includes an atomic coordinate reconstruction procedure, and its final structural outputs can contain C1′ coordinates. Could the authors clarify which DRfold representation is used as input to ResidueGNN? More importantly, have they compared ResidueGNN against directly extracting C1′ coordinates from the final structures generated by the official DRfold pipeline? This comparison is necessary to establish the independent contribution of ResidueGNN.
Q6. Positioning relative to recent methods and reproducibility. Recent methods such as DRfold2 and trRosettaRNA2 are particularly relevant to RNA 3D structure and conformer prediction. Although fully matched comparisons may be challenging, additional discussion or comparisons under a common evaluation protocol would help establish the significance of the proposed approach. For ATS, could the authors also provide the exact values and tuning procedure for the diversity and structural-prior coefficients, together with the computational overhead of generating and selecting the 15 candidates? Releasing the candidate structures, prediction files, and ATS selection implementation would substantially improve reproducibility.
We thank the reviewer for recognizing our empirical improvements and for the constructive feedback. We address W1–W3 and Q1–Q6 below.
W1: ATS novelty and baselines
ATS uses greedy, state-dependent selection rather than multi-branch tree search. It selects five of 15 Protenix/TBM/Boltz candidates using source reliability, Kabsch diversity, and structural priors, updating scores after each selection.
Matched comparisons on the same candidate pool:
ATS exceeds anchored MMR by +0.0020/+0.0117. TBM-only achieves 0.570/0.530 versus ATS 0.684/0.632. Anchor removal and equal weights slightly improve private results, so reference parameters are not universally optimal. Oracle best-of-15 cannot be calculated from aggregate private scores without native structures.
W2: Homology-aware generalization
We analyzed 75 external records: YangLab30 (30), RNA-Puzzles (20), CASP15 (12), and post-cutoff PDB (13).
BLAST comparisons requiring ≥80% bidirectional coverage found only 1/75 records at ≥70% identity and none at ≥95%.
US-align comparisons normalized by evaluation-reference length found 9/75 matches above TM=0.45 and 4/75 above TM=0.50 after retrospective filtering. Thus, 66/75 (88%) have no detected match above 0.45.
Rfam/Infernal identified 11 shared-family records, 40 annotated without detected overlap, and 24 unassigned. Among annotated records, 40/51 (78.4%) have no detected shared family.
These results measure detected similarity, not evolutionary independence or novel folds. Retrospective filtering does not alter trained checkpoints. Details appear in Appendix N.1. Prediction stratification by similarity and length remains desirable.
W3/Q1: Private evaluation
Original models, ATS weights, and anchors were selected using public feedback and local validation before future private evaluation. Organizers evaluated fixed submissions on future targets unavailable during development.
Additional ablations were retrospective, following release of aggregate private scores, and did not influence original configurations.
Q2: MSA-free inference
Organizers supplied precomputed MSAs. We tested them but observed no meaningful accuracy improvement, while loading and featurization increased overhead. MSA-free inference was selected under P100 runtime constraints, while training remained MSA-assisted.
Q3: Multi-conformation supervision
Experimental coordinates were grouped by sequence; multiple records may represent chains, models, or deposits rather than distinct biological states.
Alternatives were ordered by C1′ TM-score relative to a fixed primary reference. The objective is:
L_multi = min_k L(pred, GT_k) + 0.1 L(pred, GT_1).Supervision improves TM from 0.420/0.381 to 0.431/0.412 without MSA. It accommodates alternative references but does not explicitly maximize output diversity. Reference-pair TM-score distributions remain unreported.
Q4: Synthetic augmentation
We used public UW synthetic structures, not a generator trained by us. Of 3,982 candidates, 400 were retained through BLAST, RMSD/TM-score comparisons, and Rfam-informed selection of non-redundant examples from existing training families.
This assesses diversity, not experimental correctness. Merged-v1 achieves 0.448/0.424 versus Merged-v2 0.489/0.463, but Merged-v2 also removes approximately 40 CASP16-derived sequences. A controlled augmentation-only experiment remains necessary.
Q5: DRfold2 and ResidueGNN
ResidueGNN converts intermediate DRfold2 MSA2XYZ coordinates (P, C4′, N1/N9) directly into C1′ coordinates.
Official DRfold2 can reconstruct C1′, but we avoided full reconstruction/refinement due to computational and dependency overhead in the offline P100 environment. ResidueGNN is a separate baseline, not part of ATS. Direct comparison with official reconstruction remains necessary.
Q6: Coefficients and reproducibility
We expanded discussion of DRfold2 and trRosettaRNA2; matched comparisons remain desirable.
ATS uses source weights (0.75, 0.15, 0.10), diversity coefficient 1.0, prior penalty 0.5, and anchors Protenix #2 + TBM #1.
Score(c|S) = w_model(c)[1.0 D(c,S) - 0.5 P(c)]Protenix/Boltz generation dominates runtime; ATS adds limited CPU overhead. We intend to release implementation details and permitted artifacts.
We believe these additions strengthen the methodology and evaluation and respectfully invite reconsideration.