- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines disagreement between ESMFold and MSA-based structure predictors on artificial two-domain fusion proteins. The authors construct 1,000 chimeric domain pairs and evaluate preservation of the constituent folds against AFDB reference conformations. Comparisons with Boltz-2, AF2, and additional controls show that ESMFold produces more low-scoring predictions and exhibits domain-order preferences that do not transfer consistently across predictors. Further analyses distinguish isolated-domain prediction difficulty from fusion-dependent losses, examine pair-representation changes, and introduce a dummy-fusion screen for identifying context-sensitive domains.
Critical strengths
-
A relevant evaluation question supported by extensive controls. The reliability of structure predictors as computational screening tools is important for protein design. The authors examine multiple predictors, alternative MSA procedures, linker variations, domain boundaries, and available experimental references. The independent redraw and clustered uncertainty estimates strengthen the reproducibility of the observed disagreement.
-
An informative separation of failure regimes. Distinguishing domains that ESMFold already predicts poorly in isolation from domains affected specifically by fusion context improves the analysis beyond aggregate model comparisons. Recurrence across unrelated partners provides additional evidence that some context sensitivity is associated with particular domains.
Critical weaknesses
-
The demonstrated contribution is too limited to establish a substantial advance in protein-design evaluation. The paper convincingly documents disagreement among predictors, but does not establish which predictions are correct for the artificial fusion constructs. Most evaluations use AFDB conformations as references, and the additional MSA-based predictors share evolutionary inputs. The deposited-structure analysis supports recovery of known source-domain folds, but does not validate those folds within the artificial fusions. Consequently, neither lower AFDB-referenced failure rates nor stronger agreement among MSA models establishes a more reliable design-screening procedure. The authors appropriately acknowledge this distinction; nevertheless, it leaves the operational recommendations without direct validation. The principal result remains a predictor-disagreement audit in a restricted construction setting, with insufficient evidence that it changes experimentally grounded design decisions.
-
The extensions do not sufficiently overcome the descriptive nature of the study, and reproducibility is incomplete. Pair-representation drift localizes a correlated change but does not establish the mechanism responsible for context-dependent prediction loss. The dummy-fusion screen predicts a model-defined failure event, is evaluated using a single dummy partner, and reports perfect precision on a small, conditional subset. Its robustness and broader utility therefore remain uncertain. Moreover, the supplementary archive explicitly omits analysis scripts. Since the study relies on custom construction, mapping, scoring, statistical resampling, and representation extraction, result tables alone are insufficient for transparent reproduction or reuse. Taken together, these limitations leave neither a sufficiently developed diagnostic method nor a fully reproducible benchmark contribution.
Questions for authors
-
Practical significance. Beyond documenting disagreement, what validated design decision can be improved using the findings? In particular, what supports the recommendation involving joint MSA-model rejection beyond its lower rejection rate?
-
Reference dependence. How should readers distinguish preservation of an AFDB source-domain conformation from structural correctness in the artificial fusion? Please clarify which conclusions remain supported without treating either the reference conformation or predictor consensus as fusion-level ground truth.
-
Unseen-combination claims. Were assembled sequences or family combinations checked against broader UniProt collections, or only against the natural source pool? Please align claims about absence from UniProt and training-distribution novelty with the checks actually performed.
-
Diagnostic validation. How sensitive is the dummy-fusion screen to the selected partner? Please clarify the population used for its precision and recall estimates and distinguish prospective screening performance from performance conditional on an already identified predictor-disagreement subset.
-
Reproducibility. Can the authors provide the construction and analysis scripts, exact inference configurations, and commands required to reproduce the main results, including domain mappings, TM-score calculations, clustered resampling, and representation-drift measurements?