- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates in-frame indel pathogenicity prediction using conformational ensembles generated by BioEmu. Six structure-informed PLMs are evaluated as frozen feature extractors, with predictions averaged across conformations. The approach generally outperforms single-structure inputs, improving ESM3 AUROC from 0.8794 to 0.9359. The authors further analyze how conformational diversity affects protein representations and identify structural characteristics associated with improved predictions.
Critical strengths
1. Consistent improvements across multiple PLMs.
The authors evaluate six structure-informed PLMs with different architectures and structural representations, demonstrating relatively consistent improvements from conformational ensembles. The use of a common classifier architecture and multiple random seeds strengthens the empirical comparison, suggesting that the observed benefits are not specific to a single model.
2. Comprehensive empirical analysis.
The paper examines different structural inputs, ensemble sizes, representation characteristics, and prediction consistency. Additional analyses of pLDDT, residue contacts, and conformational diversity help characterize where the improvements occur. In particular, the finding that improvements are concentrated in flexible, low-confidence regions provides a plausible biological explanation for the observed performance gains.
Critical weaknesses
1. Limited novelty and substantial overlap with DynamicsPLM.
The proposed approach combines existing BioEmu and PLMs with simple prediction averaging, without introducing a new model architecture, learning objective, or aggregation mechanism. DynamicsPLM (Kalifa et al., Learning Protein Representations with Conformational Dynamics), available as a bioRxiv preprint since October 6, 2025, already explored generating conformational ensembles, converting them into structural tokens, and incorporating this information into PLM representations. Although the aggregation mechanisms and downstream tasks differ, the core technical pipeline substantially overlaps. The present work primarily extends this general approach to in-frame indel pathogenicity prediction, rather than establishing a fundamentally new methodology. Furthermore, DynamicsPLM is neither cited nor discussed, making it difficult to assess the precise contribution relative to existing work. This limited methodological novelty is my primary reason for recommending rejection.
2. Insufficient evidence of practical advantages over existing predictors.
Although CADD and INDELi-E are included, other established indel predictors, such as VEST-Indel and MetaRNN-indel, are not evaluated. A broader comparison against specialized predictors would help establish whether the additional computational cost of generating conformational ensembles is justified by meaningful predictive improvements. Moreover, Figure 5 shows that 96.5% of consistently improved variants are benign, suggesting that the observed gains may be driven primarily by improved recognition of benign variants rather than better detection of pathogenic variants. Although the overall AUROC improvements are encouraging, their practical significance for pathogenic variant identification remains insufficiently demonstrated.
Questions for authors
1. What substantially distinguishes this work from DynamicsPLM?
Beyond the differences in aggregation strategy and downstream task, what genuinely new methodological or empirical contribution does this work offer? I currently see no substantial methodological advance over DynamicsPLM. A convincing answer to this question is essential for reconsidering my assessment.
2. Could the authors include additional specialized baselines?
A comparison with one or two representative predictors, such as MetaRNN-indel or VEST-Indel, on a common scorable test subset would help establish the approach's practical competitiveness.
3. Do conformational ensembles provide meaningful dynamic information beyond reducing structural prediction variability?
It remains unclear whether the improvements arise from biologically meaningful conformational dynamics or simply from averaging out variability in structural predictions. Could the authors compare BioEmu ensembles against an ensemble of 10 AlphaFold3 structures generated with different random seeds, using the same prediction-averaging strategy? Such a control would help establish whether BioEmu captures additional biologically relevant information beyond the benefits of repeated structure prediction and ensemble averaging.
4 comments
We thank the reviewer for the suggestion to clarify the distinctions between our work and DynamicsPLM. We agree that the ingredients constituting the core technical pipeline might overlap. Addressing this comment, together with the newly added results, has helped us clearly articulate the distinct contributions of our approach along two key dimensions. We discussed these points in the revised manuscript.
First, we found that our framework effectively leverages information from conformational ensembles, whereas DynamicsPLM remains largely confined within the representation distribution of static structures predicted by AlphaFold2 (AF2). Second, our methodology is readily extensible across diverse structure-informed protein language models (PLMs) and adaptation strategies (e.g., zero-shot, probing, and fine-tuning). In contrast, extending DynamicsPLM to other backbones requires developing a new methodology tailored to each specific PLM, whereas its applicability across adaptation strategies remains undemonstrated.
Leveraging information from conformational ensembles. The newly added analysis of PLM's output representation indicates that even when constructed from the exact same BioEmu conformational ensemble, the output embeddings of DynamicsPLM remain largely confined as a sub-distribution within the embedding space of AlphaFold2 static structures (generated across different random seeds). In contrast, our approach yields a distinct distribution that separates ensemble dynamics from static structures (Appendix V). Given our finding that capturing conformational distributions beyond static predictions is important for variant interpretation, DynamicsPLM’s confinement within the static-structure embedding space suggests that DynamicsPLM may struggle to process conformational ensemble information through the protein language model for downstream variant tasks. Consistent with this observation, DynamicsPLM not only shows significantly lower performance than our method on both fine-tuned benign/pathogenic classification and zero-shot DMS benchmarks, but also achieves results comparable to those obtained using individual static structures (Appendix P and U).
Extending to diverse structure-informed PLMs and adaptation strategies. Our methodology introduces no changes to the pre-trained input format and imposes no special constraints. Consequently, it can be applied identically to most structure-informed PLMs and readily extended to different adaptation strategies. In contrast, extending DynamicsPLM to other backbones requires developing a new methodology tailored to each specific PLM. Specifically, S2F represents protein structures as spatial graphs based on alpha-carbon coordinates rather than discrete tokens; while node features can be averaged across conformers, shifts in spatial proximity alter the graph connections, preventing the direct input-level mixing proposed in DynamicsPLM. For models with large structural vocabularies, such as ProSST and VenusREM (2,048 tokens) or ESM3 (4,096 tokens), the structural space is divided so finely that even minor thermal fluctuations can cause residues to switch tokens frequently. Because a limited set of conformers is mapped onto thousands of possible tokens, the resulting distributions become sparse and overly sensitive to pseudocount settings, risking either sampling bias or over-smoothing. Furthermore, ESM3 incorporates additional structural tracks (3D coordinates, SS8, and SASA) alongside discrete tokens, precluding straightforward token-only mixing. DynamicsPLM converts discrete tokens into soft tokens, using a novel input format never seen during training, which makes it unclear whether meaningful information can be conveyed in zero-shot tasks without additional fine-tuning.
Finally, while we fully understand the reviewer's perspective regarding downstream tasks, we respectfully note that modeling mutation effects involves distinct challenges compared to modeling wild-type (WT) proteins alone. Whereas WT modeling characterizes native equilibrium states, predicting mutation outcomes requires capturing subtle perturbations, including changes in thermodynamic stability and conformational coupling, that are difficult to infer directly from WT data. Given its practical importance in protein engineering and disease variant interpretation, we believe that dedicated modeling of these mutation-induced perturbations remains a meaningful problem in its own right.
2. Insufficient evidence of practical advantages over existing predictors. Although CADD and INDELi-E are included, other established indel predictors, such as VEST-Indel and MetaRNN-indel, are not evaluated. A broader comparison against specialized predictors would help establish whether the additional computational cost of generating conformational ensembles is justified by meaningful predictive improvements. Moreover, Figure 5 shows that 96.5% of consistently improved variants are benign, suggesting that the observed gains may be driven primarily by improved recognition of benign variants rather than better detection of pathogenic variants. Although the overall AUROC improvements are encouraging, their practical significance for pathogenic variant identification remains insufficiently demonstrated.
2. Could the authors include additional specialized baselines? A comparison with one or two representative predictors, such as MetaRNN-indel or VEST-Indel, on a common scorable test subset would help establish the approach's practical competitiveness.
We added two further analyses. First, we incorporated comparisons against VEST-Indel and MetaRNN-indel. Because both tools were trained on clinical variant data, we conducted a chronological evaluation on variants deposited from 2025 onward to avoid data circularity (Appendix R). Although meta-predictors such as MetaRNN-indel integrating multiple tools are generally known to outperform single models, PLMs fine-tuned with our framework achieved higher performance than MetaRNN-indel. Second, to verify the practical significance of our predictions for pathogenic variant identification, we assessed the clinical evidence strength achieved across both insertions and deletions. PLMs fine-tuned with our framework achieved the PP3_Strong (+4) evidence threshold for both insertions and deletions, whereas existing tools reach at most +3 (Appendix S). Specifically, we calibrated our scores using the ClinGen local posterior probability framework (Pejaver et al., 2022) as applied to in-frame indels by Abderrazzaq et al. (2026). Prior to calibration, we retrained PLMs after excluding all variants in their calibration set (ClinVar 2023) and evaluation set (ClinVar 2025) from our training data. Within this Bayesian framework, evidence points scale exponentially on a log-odds scale, where each single-point increment requires an approximately 2.08-fold increase in the odds of pathogenicity relative to the prior. Consequently, reaching the PP3_Strong (+4) threshold requires an ~18.7-fold enrichment in pathogenic odds over the prior—more than double the ~9.0-fold enrichment required at the +3 level. PLMs fine-tuned with our framework successfully reached this previously unattained PP3_Strong (+4) evidence strength threshold for both insertions and deletions, whereas none of the existing tools evaluated by Abderrazzaq et al. achieved this level (reaching at most +3). Thus, the performance gains are not limited to benign variants; PLMs fine-tuned with our framework achieve Strong-level pathogenic evidence that existing tools fail to reach.
3. Do conformational ensembles provide meaningful dynamic information beyond reducing structural prediction variability?
It remains unclear whether the improvements arise from biologically meaningful conformational dynamics or simply from averaging out variability in structural predictions. Could the authors compare BioEmu ensembles against an ensemble of 10 AlphaFold3 structures generated with different random seeds, using the same prediction-averaging strategy? Such a control would help establish whether BioEmu captures additional biologically relevant information beyond the benefits of repeated structure prediction and ensemble averaging.
We thank the reviewer for raising this important distinction. Our additional controls show that the performance gains arise from biologically meaningful conformational dynamics rather than from averaging out variability in repeated structure predictions. Using the exact same prediction-averaging strategy, we compared BioEmu ensembles against two controls (Appendix L). The first is an ensemble of up to 50 AF2 structures generated by varying only the random seed without MSA subsampling, a setting shown to capture prediction variability without sampling true conformational ensembles, unlike shallow MSA subsampling (Del Alamo et al., 2022). We used AF2 instead of AF3 because of licensing restrictions on the AF3 model parameters, and because our single-structure baselines also use AF2 predictions, which keeps this control directly comparable to them. The second control adds structural noise to the BioEmu ensembles. BioEmu ensembles steadily improved predictive performance as the ensemble size increased, whereas aggregating multi-seed AF2 structures yielded no gain over a single structure. Adding noise to the BioEmu structures progressively diminished the ensemble-driven improvement. Together, these results indicate that high-quality conformational ensembles provide dynamic information relevant to indel pathogenicity, beyond the benefits of repeated structure prediction and ensemble averaging.
We additionally confirmed that variant-specific conformational ensembles are important for predicting variant pathogenicity. We observed that mutant conformational ensembles capture additional dynamic information not present in the wild type (WT). In a validation across four PLMs with smaller embedding sizes, combining both the WT and variant ensembles achieved higher predictive performance than using the same total number of conformers from the WT alone (Appendix J). This finding suggests that variant ensembles provide unique structural signals absent from WT ensembles. To evaluate this systematically, we designed an experimental setup in which the static WT structure, the WT conformational ensemble, and the indel conformational ensemble were provided together. To ensure an equal total number of conformers during training, we compared a setting that fed two distinct subsets of five WT conformers into the two ensemble tracks against a setting that fed five WT conformers and five variant conformers.
When using the WT conformational ensemble alone, we observed statistically significant improvements in pathogenicity prediction performance across all six PLMs compared to using a single static structure. Furthermore, combining both the WT and variant ensembles yielded significant additional performance gains in four of the six PLMs (Appendix H and I). Notably, for pathogenic variants whose predictions improved when incorporating both ensembles, the KL divergence of the variant ensemble relative to the WT ensemble was greater than for those without improvement (Appendix K). Thus, while the WT ensemble provides informative structural context for variant interpretation, the mutant specific ensemble offers additional information that the WT ensemble alone cannot capture.
All changes are additive. No result from the submitted version has been removed or altered, and new material is marked in the revised PDF that is now uploaded on OpenReview.
References
Del Alamo, Diego, et al. "Sampling alternative conformational states of transporters and receptors with AlphaFold2." elife 11 (2022): e75751.
We sincerely thank you for your insightful review. Through these suggested experiments, we could establish three supporting observations that substantiate our main claim. First, our findings suggest that our approach captures ensemble dynamics beyond static AlphaFold2 representations more effectively than DynamicsPLM, and can be readily applied across various PLMs and adaptation strategies without modifying the models. Second, on a chronological post-2025 clinical benchmark, PLMs fine-tuned with our framework outperform established specialized and meta-predictors (including VEST-Indel and MetaRNN-indel) and uniquely achieve the previously unattained PP3_Strong (+4) clinical evidence threshold for both insertions and deletions, demonstrating clear practical utility for pathogenic variant identification. Third, AlphaFold2 seed controls and noise perturbation experiments confirm that our gains stem from biologically meaningful conformational dynamics rather than simple variance reduction across repeated predictions.
Below are our detailed responses to each of your comments. To deliver our response concisely and clearly, we have grouped related weaknesses and questions together. All newly added figures presenting the corresponding results in the revised manuscript are annotated with their respective section references.
1. Limited novelty and substantial overlap with DynamicsPLM. The proposed approach combines existing BioEmu and PLMs with simple prediction averaging, without introducing a new model architecture, learning objective, or aggregation mechanism. DynamicsPLM (Kalifa et al., Learning Protein Representations with Conformational Dynamics), available as a bioRxiv preprint since October 6, 2025, already explored generating conformational ensembles, converting them into structural tokens, and incorporating this information into PLM representations. Although the aggregation mechanisms and downstream tasks differ, the core technical pipeline substantially overlaps. The present work primarily extends this general approach to in-frame indel pathogenicity prediction, rather than establishing a fundamentally new methodology. Furthermore, DynamicsPLM is neither cited nor discussed, making it difficult to assess the precise contribution relative to existing work. This limited methodological novelty is my primary reason for recommending rejection.
1. What substantially distinguishes this work from DynamicsPLM? Beyond the differences in aggregation strategy and downstream task, what genuinely new methodological or empirical contribution does this work offer? I currently see no substantial methodological advance over DynamicsPLM. A convincing answer to this question is essential for reconsidering my assessment.