acceptodds
Under review as a conference paper at ICLR 2027

One Structure Is Not Enough: Conformational Ensemble Representations Improve In-Frame Indel Effect Prediction Across Protein Language Models

Abstract

Predicting the functional effect of in-frame insertions and deletions (indels) is essential in genome interpretation for clinical diagnosis, genomic research and drug development. Structural information is one of the main sources for variant effect prediction, as variant effects are often mediated by protein structure alteration. A predicted static wild-type structure has enabled structure-informed protein language models (PLMs) to achieve high accuracy in genomic variant effect prediction, especially for missense variants. However, current computational tools show relatively lower prediction performance for in-frame indels, which are also difficult for experts to interpret manually. This limitation may partly stem from relying on a single static structure, since (i) AlphaFold’s reliability for structure-disrupting variants remains uncertain, and (ii) a static structure cannot fully capture the protein dynamics linked to indel tolerance, such as local flexibility. Here, we systematically evaluate conformational ensembles generated from indel variant sequences as structural input to six structure-informed PLMs used as frozen feature extractors. Across 4,270 in-frame indels from 1,418 proteins, the PLMs’ pathogenicity prediction improved significantly with the ensembles over single static structure inputs in most comparisons, especially for ESM3, whose AUROC rose from 0.8794 to 0.9359. Most PLMs also produced more variant-specific representations, and the improvement was concentrated in regions with low pLDDT, unstructured regions, and long benign variants. Consistent with recent findings on the relationship between intrinsic structural dynamics and protein tolerance to indels, these results suggest a way to incorporate such biological knowledge into PLMs at the representation level.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
In-frame indels, Protein Language Models, Conformational ensembles, Variant effect prediction, Structural dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ba5mOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates in-frame indel pathogenicity prediction using conformational ensembles generated by BioEmu. Six structure-informed PLMs are evaluated as frozen feature extractors, with predictions averaged across conformations. The approach generally outperforms single-structure inputs, improving ESM3 AUROC from 0.8794 to 0.9359. The authors further analyze how conformational diversity affects protein representations and identify structural characteristics associated with improved predictions.

Critical strengths

1. Consistent improvements across multiple PLMs.

The authors evaluate six structure-informed PLMs with different architectures and structural representations, demonstrating relatively consistent improvements from conformational ensembles. The use of a common classifier architecture and multiple random seeds strengthens the empirical comparison, suggesting that the observed benefits are not specific to a single model.

2. Comprehensive empirical analysis.

The paper examines different structural inputs, ensemble sizes, representation characteristics, and prediction consistency. Additional analyses of pLDDT, residue contacts, and conformational diversity help characterize where the improvements occur. In particular, the finding that improvements are concentrated in flexible, low-confidence regions provides a plausible biological explanation for the observed performance gains.

Critical weaknesses

1. Limited novelty and substantial overlap with DynamicsPLM.

The proposed approach combines existing BioEmu and PLMs with simple prediction averaging, without introducing a new model architecture, learning objective, or aggregation mechanism. DynamicsPLM (Kalifa et al., Learning Protein Representations with Conformational Dynamics), available as a bioRxiv preprint since October 6, 2025, already explored generating conformational ensembles, converting them into structural tokens, and incorporating this information into PLM representations. Although the aggregation mechanisms and downstream tasks differ, the core technical pipeline substantially overlaps. The present work primarily extends this general approach to in-frame indel pathogenicity prediction, rather than establishing a fundamentally new methodology. Furthermore, DynamicsPLM is neither cited nor discussed, making it difficult to assess the precise contribution relative to existing work. This limited methodological novelty is my primary reason for recommending rejection.

2. Insufficient evidence of practical advantages over existing predictors.

Although CADD and INDELi-E are included, other established indel predictors, such as VEST-Indel and MetaRNN-indel, are not evaluated. A broader comparison against specialized predictors would help establish whether the additional computational cost of generating conformational ensembles is justified by meaningful predictive improvements. Moreover, Figure 5 shows that 96.5% of consistently improved variants are benign, suggesting that the observed gains may be driven primarily by improved recognition of benign variants rather than better detection of pathogenic variants. Although the overall AUROC improvements are encouraging, their practical significance for pathogenic variant identification remains insufficiently demonstrated.

Questions for authors

1. What substantially distinguishes this work from DynamicsPLM?

Beyond the differences in aggregation strategy and downstream task, what genuinely new methodological or empirical contribution does this work offer? I currently see no substantial methodological advance over DynamicsPLM. A convincing answer to this question is essential for reconsidering my assessment.

2. Could the authors include additional specialized baselines?

A comparison with one or two representative predictors, such as MetaRNN-indel or VEST-Indel, on a common scorable test subset would help establish the approach's practical competitiveness.

3. Do conformational ensembles provide meaningful dynamic information beyond reducing structural prediction variability?

It remains unclear whether the improvements arise from biologically meaningful conformational dynamics or simply from averaging out variability in structural predictions. Could the authors compare BioEmu ensembles against an ensemble of 10 AlphaFold3 structures generated with different random seeds, using the same prediction-averaging strategy? Such a control would help establish whether BioEmu captures additional biologically relevant information beyond the benefits of repeated structure prediction and ensemble averaging.

4 comments

We sincerely thank you for your insightful review. Through these suggested experiments, we could establish three supporting observations that substantiate our main claim. First, our findings suggest that our approach captures ensemble dynamics beyond static AlphaFold2 representations more effectively than DynamicsPLM, and can be readily applied across various PLMs and adaptation strategies without modifying the models. Second, on a chronological post-2025 clinical benchmark, PLMs fine-tuned with our framework outperform established specialized and meta-predictors (including VEST-Indel and MetaRNN-indel) and uniquely achieve the previously unattained PP3_Strong (+4) clinical evidence threshold for both insertions and deletions, demonstrating clear practical utility for pathogenic variant identification. Third, AlphaFold2 seed controls and noise perturbation experiments confirm that our gains stem from biologically meaningful conformational dynamics rather than simple variance reduction across repeated predictions.

Below are our detailed responses to each of your comments. To deliver our response concisely and clearly, we have grouped related weaknesses and questions together. All newly added figures presenting the corresponding results in the revised manuscript are annotated with their respective section references.

1. Limited novelty and substantial overlap with DynamicsPLM. The proposed approach combines existing BioEmu and PLMs with simple prediction averaging, without introducing a new model architecture, learning objective, or aggregation mechanism. DynamicsPLM (Kalifa et al., Learning Protein Representations with Conformational Dynamics), available as a bioRxiv preprint since October 6, 2025, already explored generating conformational ensembles, converting them into structural tokens, and incorporating this information into PLM representations. Although the aggregation mechanisms and downstream tasks differ, the core technical pipeline substantially overlaps. The present work primarily extends this general approach to in-frame indel pathogenicity prediction, rather than establishing a fundamentally new methodology. Furthermore, DynamicsPLM is neither cited nor discussed, making it difficult to assess the precise contribution relative to existing work. This limited methodological novelty is my primary reason for recommending rejection.

1. What substantially distinguishes this work from DynamicsPLM? Beyond the differences in aggregation strategy and downstream task, what genuinely new methodological or empirical contribution does this work offer? I currently see no substantial methodological advance over DynamicsPLM. A convincing answer to this question is essential for reconsidering my assessment.

We thank the reviewer for the suggestion to clarify the distinctions between our work and DynamicsPLM. We agree that the ingredients constituting the core technical pipeline might overlap. Addressing this comment, together with the newly added results, has helped us clearly articulate the distinct contributions of our approach along two key dimensions. We discussed these points in the revised manuscript.

First, we found that our framework effectively leverages information from conformational ensembles, whereas DynamicsPLM remains largely confined within the representation distribution of static structures predicted by AlphaFold2 (AF2). Second, our methodology is readily extensible across diverse structure-informed protein language models (PLMs) and adaptation strategies (e.g., zero-shot, probing, and fine-tuning). In contrast, extending DynamicsPLM to other backbones requires developing a new methodology tailored to each specific PLM, whereas its applicability across adaptation strategies remains undemonstrated.

Leveraging information from conformational ensembles. The newly added analysis of PLM's output representation indicates that even when constructed from the exact same BioEmu conformational ensemble, the output embeddings of DynamicsPLM remain largely confined as a sub-distribution within the embedding space of AlphaFold2 static structures (generated across different random seeds). In contrast, our approach yields a distinct distribution that separates ensemble dynamics from static structures (Appendix V). Given our finding that capturing conformational distributions beyond static predictions is important for variant interpretation, DynamicsPLM’s confinement within the static-structure embedding space suggests that DynamicsPLM may struggle to process conformational ensemble information through the protein language model for downstream variant tasks. Consistent with this observation, DynamicsPLM not only shows significantly lower performance than our method on both fine-tuned benign/pathogenic classification and zero-shot DMS benchmarks, but also achieves results comparable to those obtained using individual static structures (Appendix P and U).

Extending to diverse structure-informed PLMs and adaptation strategies. Our methodology introduces no changes to the pre-trained input format and imposes no special constraints. Consequently, it can be applied identically to most structure-informed PLMs and readily extended to different adaptation strategies. In contrast, extending DynamicsPLM to other backbones requires developing a new methodology tailored to each specific PLM. Specifically, S2F represents protein structures as spatial graphs based on alpha-carbon coordinates rather than discrete tokens; while node features can be averaged across conformers, shifts in spatial proximity alter the graph connections, preventing the direct input-level mixing proposed in DynamicsPLM. For models with large structural vocabularies, such as ProSST and VenusREM (2,048 tokens) or ESM3 (4,096 tokens), the structural space is divided so finely that even minor thermal fluctuations can cause residues to switch tokens frequently. Because a limited set of conformers is mapped onto thousands of possible tokens, the resulting distributions become sparse and overly sensitive to pseudocount settings, risking either sampling bias or over-smoothing. Furthermore, ESM3 incorporates additional structural tracks (3D coordinates, SS8, and SASA) alongside discrete tokens, precluding straightforward token-only mixing. DynamicsPLM converts discrete tokens into soft tokens, using a novel input format never seen during training, which makes it unclear whether meaningful information can be conveyed in zero-shot tasks without additional fine-tuning.

Finally, while we fully understand the reviewer's perspective regarding downstream tasks, we respectfully note that modeling mutation effects involves distinct challenges compared to modeling wild-type (WT) proteins alone. Whereas WT modeling characterizes native equilibrium states, predicting mutation outcomes requires capturing subtle perturbations, including changes in thermodynamic stability and conformational coupling, that are difficult to infer directly from WT data. Given its practical importance in protein engineering and disease variant interpretation, we believe that dedicated modeling of these mutation-induced perturbations remains a meaningful problem in its own right.

2. Insufficient evidence of practical advantages over existing predictors. Although CADD and INDELi-E are included, other established indel predictors, such as VEST-Indel and MetaRNN-indel, are not evaluated. A broader comparison against specialized predictors would help establish whether the additional computational cost of generating conformational ensembles is justified by meaningful predictive improvements. Moreover, Figure 5 shows that 96.5% of consistently improved variants are benign, suggesting that the observed gains may be driven primarily by improved recognition of benign variants rather than better detection of pathogenic variants. Although the overall AUROC improvements are encouraging, their practical significance for pathogenic variant identification remains insufficiently demonstrated.

2. Could the authors include additional specialized baselines? A comparison with one or two representative predictors, such as MetaRNN-indel or VEST-Indel, on a common scorable test subset would help establish the approach's practical competitiveness.

We added two further analyses. First, we incorporated comparisons against VEST-Indel and MetaRNN-indel. Because both tools were trained on clinical variant data, we conducted a chronological evaluation on variants deposited from 2025 onward to avoid data circularity (Appendix R). Although meta-predictors such as MetaRNN-indel integrating multiple tools are generally known to outperform single models, PLMs fine-tuned with our framework achieved higher performance than MetaRNN-indel. Second, to verify the practical significance of our predictions for pathogenic variant identification, we assessed the clinical evidence strength achieved across both insertions and deletions. PLMs fine-tuned with our framework achieved the PP3_Strong (+4) evidence threshold for both insertions and deletions, whereas existing tools reach at most +3 (Appendix S). Specifically, we calibrated our scores using the ClinGen local posterior probability framework (Pejaver et al., 2022) as applied to in-frame indels by Abderrazzaq et al. (2026). Prior to calibration, we retrained PLMs after excluding all variants in their calibration set (ClinVar 2023) and evaluation set (ClinVar 2025) from our training data. Within this Bayesian framework, evidence points scale exponentially on a log-odds scale, where each single-point increment requires an approximately 2.08-fold increase in the odds of pathogenicity relative to the prior. Consequently, reaching the PP3_Strong (+4) threshold requires an ~18.7-fold enrichment in pathogenic odds over the prior—more than double the ~9.0-fold enrichment required at the +3 level. PLMs fine-tuned with our framework successfully reached this previously unattained PP3_Strong (+4) evidence strength threshold for both insertions and deletions, whereas none of the existing tools evaluated by Abderrazzaq et al. achieved this level (reaching at most +3). Thus, the performance gains are not limited to benign variants; PLMs fine-tuned with our framework achieve Strong-level pathogenic evidence that existing tools fail to reach.

3. Do conformational ensembles provide meaningful dynamic information beyond reducing structural prediction variability?

It remains unclear whether the improvements arise from biologically meaningful conformational dynamics or simply from averaging out variability in structural predictions. Could the authors compare BioEmu ensembles against an ensemble of 10 AlphaFold3 structures generated with different random seeds, using the same prediction-averaging strategy? Such a control would help establish whether BioEmu captures additional biologically relevant information beyond the benefits of repeated structure prediction and ensemble averaging.

We thank the reviewer for raising this important distinction. Our additional controls show that the performance gains arise from biologically meaningful conformational dynamics rather than from averaging out variability in repeated structure predictions. Using the exact same prediction-averaging strategy, we compared BioEmu ensembles against two controls (Appendix L). The first is an ensemble of up to 50 AF2 structures generated by varying only the random seed without MSA subsampling, a setting shown to capture prediction variability without sampling true conformational ensembles, unlike shallow MSA subsampling (Del Alamo et al., 2022). We used AF2 instead of AF3 because of licensing restrictions on the AF3 model parameters, and because our single-structure baselines also use AF2 predictions, which keeps this control directly comparable to them. The second control adds structural noise to the BioEmu ensembles. BioEmu ensembles steadily improved predictive performance as the ensemble size increased, whereas aggregating multi-seed AF2 structures yielded no gain over a single structure. Adding noise to the BioEmu structures progressively diminished the ensemble-driven improvement. Together, these results indicate that high-quality conformational ensembles provide dynamic information relevant to indel pathogenicity, beyond the benefits of repeated structure prediction and ensemble averaging.

We additionally confirmed that variant-specific conformational ensembles are important for predicting variant pathogenicity. We observed that mutant conformational ensembles capture additional dynamic information not present in the wild type (WT). In a validation across four PLMs with smaller embedding sizes, combining both the WT and variant ensembles achieved higher predictive performance than using the same total number of conformers from the WT alone (Appendix J). This finding suggests that variant ensembles provide unique structural signals absent from WT ensembles. To evaluate this systematically, we designed an experimental setup in which the static WT structure, the WT conformational ensemble, and the indel conformational ensemble were provided together. To ensure an equal total number of conformers during training, we compared a setting that fed two distinct subsets of five WT conformers into the two ensemble tracks against a setting that fed five WT conformers and five variant conformers.

When using the WT conformational ensemble alone, we observed statistically significant improvements in pathogenicity prediction performance across all six PLMs compared to using a single static structure. Furthermore, combining both the WT and variant ensembles yielded significant additional performance gains in four of the six PLMs (Appendix H and I). Notably, for pathogenic variants whose predictions improved when incorporating both ensembles, the KL divergence of the variant ensemble relative to the WT ensemble was greater than for those without improvement (Appendix K). Thus, while the WT ensemble provides informative structural context for variant interpretation, the mutant specific ensemble offers additional information that the WT ensemble alone cannot capture.

All changes are additive. No result from the submitted version has been removed or altered, and new material is marked in the revised PDF that is now uploaded on OpenReview.

References

Del Alamo, Diego, et al. "Sampling alternative conformational states of transporters and receptors with AlphaFold2." elife 11 (2022): e75751.

Reviewer_m9CrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to use an ensemble of predicted structures (conformers) generated with BioEmu to create ensemble predictions to classify indels as benign or pathogenic (binary classification). The papers contribution is the ensemble and the effect is validated on a dataset 4,270 indels under a similarity-based split. The 10-conformation ensemble outperforms static WT, static indel and single-conformation inputs in 15 of 18 comparisons.

Critical strengths

The paper is well written and quite easy to follow.

The problem is important: in-frame indels are underrepresented in variant databases and poorly predicted by current tools, and structure-informed PLMs have rarely been applied to them.

The evaluation on the model comprises multiple PLMs and the performance increase is small, but appears reliable/constant. The similarity-based split and the identical head across PLMs are good practice.

Critical weaknesses
  1. The evaluation is limited to a single dataset, problem, and metric, which limits the claims the authors can make. What is missing are i) More datasets: Validation and test sets use only ClinVar-derived variants; the Evidence-classified and gnomAD variants are used for training only. All comparisons therefore rest on one dataset. Please evaluate on additional datasets. It would have been nice to put the results in perspective by evaluating on the ProteinGym subset of indel variants. we believe the current dataset is a superset of this data, so we don’t understand why the authors did not also show results on this subset. This would have made it easier to put results into perspective. ii) Stronger comparisons: BioEmu ensembles are compared only with two static structures and a single BioEmu sample. Please add other methods and ensemble generators. Because the ensemble averages 10 predictions and the baselines none, part of the gain may be generic test-time ensembling. iii) Regarding the evaluation, AUROC alone is not enough. The test set is imbalanced (about 2,630 pathogenic vs 1,640 benign, from Figure 5J). The authors should report other metrics such as AUPRC or MCC. Since the gains are mainly on benign variants, sensitivity on pathogenic variants should also be reported. Also, we have concerns about the reliability of the Wilcoxon results. The test treats 15 runs (5 folds × 3 seeds) as independent, but seeds within a fold share a test set, so the p-values are inflated. There is no correction across the 18 comparisons, and several gains are only a few thousandths of AUROC. Based on the above, it is unclear from this manuscript how well the results presented in the manuscript generalize.
  2. The core idea has been developed previously (https://arxiv.org/pdf/2506.05596). While this was a more theoretical line of work, it appears sufficiently similar and theoretically more advanced, and so this reduces the novelty of this work considerably. We think this would also need to be cited and clarified on how the current work differs from this paper.
Questions for authors

It wasn’t entirely clear to us how the pooled feature was derived. Was this the only feature the authors looked at and was there any motivation for choosing exactly this representation?

Some values in the figures are unreadable (e.g., Figure 4 cell, Figure 5J…)

Some suggestions to further improve this work: Expand this to other datasets and tasks. Compare the conformational ensemble approach to more advanced feature extraction methods. We wonder if it’s possible to quantify the variance of the generated conformational ensemble and relate this to the quality of the model, e.g. via adding more noise in the generation process. This could potentially be a way to add more novelty to the work as well. Improve the reproducibility by expanding the methods description in more detail, e.g. things like data subset sizes, balances. This last is buried in some figure, but should definitely be mentioned more clearly and expanded on. Reproducibility: there is no code release.

6 comments

We sincerely thank you for your insightful review. Through these suggested experiments, we could establish four supporting observations that substantiate our main claim. First, the predictive gains of variant conformational ensembles generalize across external benchmarks and tasks, improving performance on both the ProteinGym indel clinical subset and zero-shot Deep Mutational Scanning (DMS) assays. Second, these improvements stem from high-quality, variant-specific conformational ensembles rather than a mere test-time averaging effect. By capturing structural information beyond wild-type (WT) ensembles and generalizing across alternative ensemble generators, these ensembles enable PLMs fine-tuned with our framework to outperform strong clinical meta-predictors such as MetaRNN. Third, our results remain statistically robust under fold-level paired bootstrap testing across both AUROC and AUPRC, and extend to pathogenic variant sensitivity by achieving the ClinGen PP3_Strong (+4) evidence threshold that existing tools fail to reach. Fourth, the formulation derivation for zero-shot substitutions proposed by Frellsen et al. cannot be directly applied to the indel fine-tuning task, our primary focus. However, by adapting one of its ideas (unfolded-state corrections), we were able to add new results demonstrating further gains in zero-shot indel prediction.

Below are our detailed responses to each of your comments. All newly added figures presenting the corresponding results in the revised manuscript are annotated with their respective section references.

The evaluation is limited to a single dataset, problem, and metric, which limits the claims the authors can make. What is missing are i) More datasets: Validation and test sets use only ClinVar-derived variants; the Evidence-classified and gnomAD variants are used for training only. All comparisons therefore rest on one dataset. Please evaluate on additional datasets. It would have been nice to put the results in perspective by evaluating on the ProteinGym subset of indel variants. we believe the current dataset is a superset of this data, so we don’t understand why the authors did not also show results on this subset. This would have made it easier to put results into perspective.

Thank you for your constructive review. We have added performance evaluations on the ProteinGym indel clinical benchmark, which is a subset of our dataset, as well as on a new task, the ProteinGym indel Deep Mutational Scanning (DMS) zero-shot benchmark. We performed inference on the ProteinGym indel clinical dataset. To select reliable pathogenic and benign variants, we applied the same variant selection criteria as in our main analysis (ClinVar review status of at least one star). Across all PLMs, the 10-conformation ensemble achieved the highest median performance, showing significantly higher performance in 9 out of 18 comparisons. Moreover, the 10-conformation ensemble never showed significantly lower performance in any comparison (Appendix O).

Additionally, we included zero-shot evaluation results on the ProteinGym Deep Mutational Scanning (DMS) zero-shot benchmark. Even in this zero-shot task without fine-tuning, the 10-conformation ensemble demonstrated statistically significant performance improvements over the single-structure controls. Furthermore, when applying our approach, three of the PLMs outperformed the top five ranked models on the existing ProteinGym indel zero-shot benchmark. These additions further demonstrate that, beyond clinical benchmarks, variant conformational ensembles also contribute to improved variant interpretation in zero-shot tasks on DMS benchmarks (Appendix P).

ii) Stronger comparisons: BioEmu ensembles are compared only with two static structures and a single BioEmu sample. Please add other methods and ensemble generators. Because the ensemble averages 10 predictions and the baselines none,

First, we have added results based on a new ensemble generator and the WT conformational ensemble. Conformational ensembles generated using an alternative ensemble generator, AlphaFold2 (AF2) with MSA subsampling (Del Alamo, et al., 2022), also yielded performance improvements. Additionally, we confirmed that the WT conformational ensemble improves variant interpretation performance (Appendix H and I). This aligns with prior findings showing that WT conformational ensembles provide valuable information for interpreting the pathogenicity of indel variants (Banerjee et al., 2023).

Second, very crucially, we observed further performance improvements when combining both the WT and variant conformational ensembles (Appendix H and I), indicating that the WT and mutation ensembles provide distinct, compensatory information. To verify that this gain is not simply due to an increased number of conformers, we conducted a controlled comparison across four PLMs with the smallest embedding sizes using either two distinct subsets of five WT conformers or a combination of five WT and five variant conformers. Even with the total number of conformers matched, combining both WT and variant ensembles still outperformed using ten WT conformers alone (Appendix J). Moreover, for variants whose predictions improved upon incorporating the variant ensemble, the KL divergence of the variant ensemble relative to the WT ensemble was higher than for those without improvement. These results indicate that variant-specific conformational ensembles provide information that enhances variant effect prediction performance (Appendix K).

part of the gain may be generic test-time ensembling.

Additionally, to verify that these ensemble-driven improvements do not merely stem from test-time ensembling, we evaluated multiple structures generated by AlphaFold2 (AF2) without MSA subsampling (which is known to converge without generating a conformational ensemble; Del Alamo et al., 2022), as well as BioEmu conformational ensembles with added noise. Both conditions resulted in reduced predictive performance. This demonstrates that the quality of the conformational ensemble is critical for indel variant interpretation performance (Appendix L).

iii) Regarding the evaluation, AUROC alone is not enough. The test set is imbalanced (about 2,630 pathogenic vs 1,640 benign, from Figure 5J). The authors should report other metrics such as AUPRC or MCC. Since the gains are mainly on benign variants, sensitivity on pathogenic variants should also be reported. Also, we have concerns about the reliability of the Wilcoxon results. The test treats 15 runs (5 folds × 3 seeds) as independent, but seeds within a fold share a test set, so the p-values are inflated. There is no correction across the 18 comparisons, and several gains are only a few thousandths of AUROC. Based on the above, it is unclear from this manuscript how well the results presented in the manuscript generalize.

In addition to AUROC, we now report AUPRC. To address your concern regarding the non-independence of random seeds within each fold, we recalculated the p-values. Specifically, we averaged the predictions across the three seeds per fold, computed each metric per fold, and averaged the results across all five folds. We then assessed statistical significance using a paired bootstrap test (10,000 iterations) resampling proteins within each fold. The resulting p-values were adjusted across the 18 comparisons using the Benjamini–Hochberg procedure. Under this evaluation, the 10-conformation ensemble still significantly outperformed the other three inputs in 15 of 18 comparisons for AUROC and 13 of 18 for AUPRC (q < 0.05). Moreover, it did not perform significantly worse in any comparison for either metric (Figure 2 and Appendix N).

Additionally, to directly evaluate sensitivity on pathogenic variants, we calibrated our scores using the ClinGen local posterior probability framework (Pejaver et al., 2022) as applied to in-frame indels by Abderrazzaq et al. (2026). Prior to calibration, we retrained PLMs after excluding all variants in their calibration set (ClinVar 2023) and evaluation set (ClinVar 2025) from our training data. Within this Bayesian framework, evidence points scale exponentially on a log-odds scale, where each single-point increment requires an approximately 2.08-fold increase in the odds of pathogenicity relative to the prior. Consequently, reaching the PP3_Strong (+4) threshold requires an ~18.7-fold enrichment in pathogenic odds over the prior, more than double the ~9.0-fold enrichment required at the +3 level. PLMs fine-tuned with our framework successfully reached this PP3_Strong (+4) evidence strength threshold, whereas none of the existing tools evaluated by Abderrazzaq et al. achieved this level (reaching at most +3). Thus, the performance gains are not limited to benign variants; PLMs fine-tuned with our framework achieve Strong-level pathogenic evidence that existing tools fail to reach (Appendix S).

Reviewer_F4aYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether conformational ensembles improve in frame indel pathogenicity prediction compared to single predicted structures. The authors use BioEmu to generate conformations for indel variants and evaluate with PLMs with structural modality. During training, one conformation is sampled, while at inference predictions are averaged. Empirically, the ensemble approach significantly outperforms single structure inputs.

Critical strengths
  1. In frame indel prediction is relatively understudied compared to missense variants. This paper offers a very simple and useful practice without modifying the pretrained PLMs.

  2. The authors provided interesting biological observations. The analysis links prediction improvements to low confidence, low contact regions and indel tolerance.

Critical weaknesses
  1. The experiments need additional control groups. The current experiments cannot distinguish whether the gains come from mutant specific conformational information or simply averaging predictions over multiple structures. I was wondering if the authors could add two additional baselines: (i) BioEmu ensembles generated from WT sequences, and (ii) multiple AlphaFold predicted structures, using different seeds or optionally MSA subsampling. This would help clarify if mutant specific structural information is actually necessary.

  2. The mutation induced structural changes needs validation. The paper motivates the method by the limitations of static structure prediction for mutations, but does not quantify structural differences between WT and mutant predictions, either for AlphaFold or BioEmu. This is important because Zha et al. (2026) reported that BioEmu could not reliably distinguish mutation induced ensemble changes from passenger versus driver substitutions and I was wondering if the same limitations also apply to the set of indels studied in the paper.

References

  1. Zha et al. (2026). Assessing the Performance of BioEmu in Understanding Protein Dynamics.
Questions for authors

See Critical Weaknesses.

2 comments

We sincerely thank you for your insightful review. Through these suggested experiments, we could establish three supporting observations that substantiate our main claim. First, the variant conformational ensemble provides additional information for variant interpretation that the wild-type (WT) conformational ensemble does not offer. Second, the observed performance gains arise from the conformational ensemble, rather than a mere averaging effect across multiple structures. Third, mutation-induced conformational changes differentiate pathogenic from benign variants within the local structure surrounding the variant position.

Below are our detailed responses to each of your comments. All newly added figures presenting the corresponding results in the revised manuscript are annotated with their respective section references.

The experiments need additional control groups. The current experiments cannot distinguish whether the gains come from mutant specific conformational information or simply averaging predictions over multiple structures.

Through two additional experiments, we confirmed that simply averaging predictions over multiple structures is not the primary driver of these performance gains. Specifically, we performed inference using (i) AlphaFold2 (AF2) structures generated by varying only the random seed without additional MSA subsampling, and (ii) BioEmu-generated structures perturbed with added noise (Appendix L). Unlike shallow MSA subsampling, varying only the random seed without additional MSA subsampling has been reported not to generate conformational ensembles (Stein et al., 2022). While the BioEmu conformational ensemble demonstrated steady performance improvements as the number of structures increased at inference, the AF2 structures generated with varying random seeds showed no performance improvements from ensembling, and adding noise to the BioEmu structures progressively diminished the ensemble-driven performance gains. These results demonstrate that high-quality conformational ensembles genuinely contribute to improved variant pathogenicity prediction.

I was wondering if the authors could add two additional baselines: (i) BioEmu ensembles generated from WT sequences, and

We observed that mutant conformational ensembles capture additional dynamic information not present in the wild type (WT). In a validation across four PLMs with smaller embedding sizes, combining both the WT and variant ensembles achieved higher predictive performance than using the same total number of conformers from the WT alone (Appendix J). This finding suggests that variant ensembles provide unique structural signals absent from WT ensembles. To evaluate this systematically, we designed an experimental setup in which the static WT structure, the WT conformational ensemble, and the indel conformational ensemble were provided together. To ensure an equal total number of conformers during training, we compared a setting that fed two distinct subsets of five WT conformers into the two ensemble tracks against a setting that fed five WT conformers and five variant conformers.

When using the WT conformational ensemble alone, we observed statistically significant improvements in pathogenicity prediction performance across all six PLMs compared to using a single static structure. Furthermore, combining both the WT and variant ensembles yielded significant additional performance gains in four of the six PLMs (Appendix H and I). Notably, for pathogenic variants whose predictions improved when incorporating both ensembles, the KL divergence of the variant ensemble relative to the WT ensemble was greater than for those without improvement (Appendix K). Thus, while the WT ensemble provides informative structural context for variant interpretation, the mutant specific ensemble offers additional information that the WT ensemble alone cannot capture.

(ii) multiple AlphaFold predicted structures, using different seeds or optionally MSA subsampling.

Conformational ensembles generated using AF2 with shallow MSA subsampling also improved variant pathogenicity prediction performance. Previous studies have demonstrated that performing shallow MSA subsampling in AF2 generates conformational ensembles (Stein et al., 2022, Del Alamo et al., 2022, Monteiro da Silva et al., 2024). Building on these findings, we generated conformational ensembles with AF2 and confirmed that conformational ensembles consistently enhance predictive performance even when the ensemble generator is changed (Appendix H and I).

This would help clarify if mutant specific structural information is actually necessary.

We sincerely thank you for your suggestions. Through the experiments you proposed, we were able to further substantiate that mutant-specific structural information contributes to indel variant interpretation.

The mutation induced structural changes needs validation. The paper motivates the method by the limitations of static structure prediction for mutations, but does not quantify structural differences between WT and mutant predictions, either for AlphaFold or BioEmu. This is important because Zha et al. (2026) reported that BioEmu could not reliably distinguish mutation induced ensemble changes from passenger versus driver substitutions and I was wondering if the same limitations also apply to the set of indels studied in the paper.

Thank you for pointing out this relevant study, which helped us add a valuable analysis to our findings. As reported by Zha et al., the KL divergence between the WT and variant conformational ensembles across the entire protein structure showed no difference between benign and pathogenic variants. However, when examining the conformational ensembles within the local window surrounding the variant position, pathogenic variants exhibited significantly higher KL divergence. This trend remained consistent even when comparing equal numbers of variants (Appendix M). Previous studies have reported that gain-of-function variants and stable-but-inactive variants alter protein function and cause disease through local structural changes rather than global structural alterations (Gerasimavicius et al., 2022, Matreyek et al., 2018, Cagiada et al., 2021, Cagiada et al., 2023). Accordingly, our finding that the KL divergence of local conformational ensembles better distinguishes pathogenic variants overall is consistent with prior reports.

The discussion about related works is placed in Appendix X (Appendix X.1 MUTATION-INDUCED ENSEMBLE PERTURBATIONS AND LOCAL STRUCTURAL DYNAMICS).

All changes are additive. No result from the submitted version has been removed or altered, and new material is marked in the revised PDF that is now uploaded on OpenReview.

Reference

Stein, Richard A., and Hassane S. Mchaourab. "SPEACH_AF: Sampling protein ensembles and conformational heterogeneity with Alphafold2." PLoS computational biology 18.8 (2022): e1010483.

Del Alamo, Diego, et al. "Sampling alternative conformational states of transporters and receptors with AlphaFold2." elife 11 (2022): e75751.

Monteiro da Silva, Gabriel, et al. "High-throughput prediction of protein conformational distributions with subsampled AlphaFold2." Nature communications 15.1 (2024): 2464.

Gerasimavicius, Lukas, Benjamin J. Livesey, and Joseph A. Marsh. "Loss-of-function, gain-of-function and dominant-negative mutations have profoundly different effects on protein structure." Nature communications 13.1 (2022): 3895.

Matreyek, Kenneth A., et al. “Multiplex assessment of protein variant abundance by massively parallel sequencing.” Nature genetics 50.6 (2018): 874-882.

Cagiada, Matteo, et al. “Understanding the origins of loss of protein function by analyzing the effects of thousands of variants on activity and abundance.” Molecular Biology and Evolution 38.8 (2021): 3235-3246.

Cagiada, Matteo, et al. “Discovering functionally important sites in proteins.” Nature communications 14.1 (2023): 4175.

Comments on the paper

1 comment

Dear Area Chairs,

We are posting this openly so that the reviewers can read it as well.

Thank you for the reviewer assignment. All three reviews were specific and constructive, and each suggested an experiment we could run. We have run them, and this revision reports the results. Between them the reviews raised three concerns, and our answers to all three support our main claim.

First, whether the effect is real. All three reviewers independently raised this question, which is whether the reported gain reflects conformational information specific to the variant or simply the averaging of several structures. Reviewers 1 and 3 suggested repeated structure prediction as a control. Averaging up to 50 AlphaFold2 structures that differ only by random seed gave no gain at all, adding noise to the BioEmu ensembles removed the gain progressively (Figure 17), and generating ensembles with shallow MSA subsampling brought it back (Figures 13 and 14). The effect tracks ensemble quality, not the number of forward passes. Reviewer 1 also pointed us to Zha et al., and following it we found that global ensemble divergence does not separate pathogenic from benign variants, exactly as that study reported, while divergence within the local window around the variant does (Figure 18).

Second, whether it matters clinically. Reviewers 2 and 3 noted that the gains looked concentrated on benign variants. Calibrating with the ClinGen local posterior probability framework, after retraining with every calibration and evaluation variant removed, PLMs fine-tuned with our framework reach PP3_Strong (+4) for both insertions and deletions. That threshold requires roughly an 18.7 fold enrichment in pathogenic odds over the prior, against roughly 9.0 fold at +3, and none of the predictors evaluated in that calibration study reached it (Figure 24).

Third, what this work adds relative to the papers the reviewers raised. We ran both comparisons. Built from identical BioEmu ensembles, DynamicsPLM embeddings stay inside the embedding space of static AlphaFold2 structures while ours separate from it, and on our benchmarks it performs close to a single static structure (Figure 27). The paper Reviewer 2 raised also led to a new result. Its free energy derivation does not transfer to length changing variants, but its unfolded state correction does, and applying it raised zero-shot Spearman correlation for VenusREM from 0.453 to 0.547 (Figure 22). Now the discussion about related works is newly placed in Appendix X.

This problem matters to the biological and medical community. In-frame indels are a long standing blind spot in clinical interpretation, and ClinGen’s own calibration work shows that existing tools top out at supporting or moderate evidence. Laboratories therefore classify many of these variants as uncertain, which leaves the diagnosis unresolved for those patients. Generative models of conformational ensembles became fast enough for this scale only recently, which is what makes the question answerable now rather than earlier. We hope others will build on this, because the clinical need is immediate.

All changes are additive. No result from the submitted version has been removed or altered, and new material is marked in the revised PDF that is now uploaded on OpenReview.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.