- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies out-of-distribution protein fitness prediction on FLIP2 and two additional settings. It augments frozen protein language model representations with two sources of prior information: structure-aware protein representations and pooled features from an AlphaFold-style trunk that processes each sequence and its multiple-sequence alignment. A lightweight probe maps the frozen features to fitness scores.
The experiments cover several protein encoders and multiple distribution shifts, including unseen mutation positions, mutation counts, mutation types, fitness ranges, and protein backbones. The strongest gains occur on the AMYLASE position splits. The trunk features also improve some held-out-backbone splits and two evaluations outside FLIP2. The paper further separates structural and co-evolutionary contributions with feature-block and MSA-Pairformer controls. Its main conclusion is conditional: structural priors help most across protein backbones, while trunk features help most across mutation positions. No representation is best in every regime.
Critical strengths
-
The evaluation targets practical distribution shifts and covers several protein families. This matters because standard random splits can overstate fitness-prediction utility. The paper evaluates the settings that are important in iterative protein engineering, including unseen positions and unseen backbones.
-
The component analysis produces a useful scientific conclusion. The paper does not only report a best model. It shows that structure-aware encoders and trunk features help in different regimes. The MSA-Pairformer comparison further suggests that the position-shift gains are not explained only by co-evolutionary information.
Critical weaknesses
-
The main comparison is not fully matched. The pLM-only and pLM-plus-trunk rows use different feature-normalization settings. The appendix suggests that normalization does not explain the main gains, but a matched comparison would make the contribution of the trunk features easier to interpret.
-
Uncertainty is not reported consistently for the main FLIP2 results. The paper provides uncertainty for some analyses, but the main comparison mostly reports point estimates selected from several encoders and feature combinations. This does not undermine the large position-split gains, but it makes the smaller improvements less clear.
Questions for authors
-
Could the authors provide a matched pLM-only versus pLM-plus-trunk comparison using the same normalization and probe setup?
-
Could the authors report uncertainty for the main FLIP2 results, especially for the smaller gains?
-
Could the authors comment on whether FLIP2 test proteins or close homologs may have appeared in the pretraining data of the structure-aware models or trunk? A complete audit may not be possible, but the likely overlap would help interpret the held-out-backbone results.
-
If feasible, could one strong published FLIP2 baseline be evaluated in the same probe and preprocessing pipeline? This would provide a useful matched reference for the reported-number comparison.
2 comments
Matched reference of FLIP2 baselines
If feasible, could one strong published FLIP2 baseline be evaluated in the same probe and preprocessing pipeline? This would provide a useful matched reference for the reported-number comparison.
We thank the reviewer for the suggestion to report FLIP2 baselines using the same probes. We note that the original FLIP2 evaluation considers a broader range of baseline methods, including not just MLP probes but ridge regression on one-hot features and zero-shot sequence-likelihood scoring that is not applicable for trunk features. Nevertheless, to ensure a fair comparison, we therefore report the performance of the best model re-evaluated using the same probes additionally in Table C8 in Appendix.
We hope these responses and additional experiments clarify our paper’s key contributions and address your concerns. We would appreciate your considering raising your score if these points are addressed sufficiently, or following up with any remaining questions.
We would like to thank the reviewer for considering our paper provides useful scientific insights to the field. We also appreciate thoughtful feedback, and hope to address the posted questions below. We have revised the manuscript accordingly and the changes are marked in blue.
Matched comparison of feature standardization
We thank the reviewer for the suggestion for comparing different methods under the same normalization and probe schemes. We have updated Table 1 to use the same normalized MLP probes for both pLM-only and pLM with Protenix’s trunk features. Our main findings remain unchanged, particularly the substantial trunk gains on unseen mutation positions. Originally, we used unnormalized probes for pLM-only models because normalization did not consistently improve performance. In contrast, standardization was more beneficial for pLM + trunk models due to differences in feature scales from different pre-trained models. We provide further evidence in Appendix C.3 (Tables C3–C4), showing that standardization has mixed effects on pLM-only models but more consistently improves pLM-plus-trunk models. These analyses explain our original choices, while the revised Table 1 isolates the contribution of trunk features.
Uncertainty Quantification in Main Table
We have updated Table 1 to report mean performance and standard deviations across 5 seeds, with 95% confidence intervals estimated from 2,000 bootstrap resamples. Our key findings remain unchanged. On AMYLASE unseen-position splits, trunk features yield significant gains across all eight encoders, ranging from +0.071 ± 0.019 to +0.263 ± 0.033 on close-to-far and +0.075 ± 0.021 to +0.221 ± 0.035 on far-to-close. All gains have 95% bootstrap confidence intervals surpassing zero. On RHOMAX, trunk features also moderately improve mean performance across all of the base pLMs, but with greater variability (e.g., ESM-C: +0.185 ± 0.175).
Homologs in pre-training data of structure-aware models
We appreciate the reviewer’s question regarding the potential leakage of structural information from FLIP2’s held-out set into structure-aware pLMs and even Protenix. To investigate this, we further evaluated our method on the held-out wild-type rhodopsin proteins, stratifying the test proteins according to their similarity to sequences in each model’s pre-training set. Because we have access to the pre-training datasets of Protenix-v2, ESM3, DPLM2, and SaProt, we considered a test protein to be represented in a test protein to have close homologues in a model’s pre-training set if at least one training chain shared more than 95% sequence identity over 70% of its length. By this criteria, 22 of the 36 test wild types (63% of test set) overlap with at least one model’s pre-training set. These 14 “unseen” wild types with no match, we observed significantly lower correlation (See table below). This suggests that the trunk’s contribution may be limited when test sequences are entirely novel or far distant from any sequences represented in the models’ pre-training data. We’ve added this analysis in ablation study section in the revised manuscript.