acceptodds
Withdrawn from ICLR 2027

Systematic Disagreement between ESMFold and MSA-Based Structure Predictors on Chimeric Fusion Proteins

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Single-sequence structure predictors such as ESMFold are widely used as computational filters for designed proteins, including multi-domain fusions absent from natural sequence databases. Whether this filter agrees with MSA-based predictors on such constructs has not been established. We assemble 1,000 chimeric domain pairs that do not occur in UniProt, held out by protein family, and score each fusion by the worse domain's TM-score against its native AlphaFold Database conformation. ESMFold exhibits a strong, internally reproducible preference for one domain order (Pearson r=0.90 under composition-matched linker swaps) that Boltz-2 does not share (r=0.07). On an unbiased subset of 300 constructs, ESMFold assigns TM<0.7 to 17.7% of fusions and Boltz-2 to 8.7%, with only ten shared failures. AlphaFold 2, on the same alignments, fails on 7/10 of those consensus failures and on 1/43 of the failures unique to ESMFold. Isolated-domain controls partition the latter set: 26 constructs contain a domain that ESMFold already fails alone—domains that isolated Boltz-2 and AlphaFold 2 both fold—and 17 fail only after fusion. The fusion-dependent failures coincide with elevated drift in the affected domain's pair block, recur across unrelated partners, and a held-out dummy fusion flags them at high precision. On the six ESMFold-only domains with deposited crystal structures, Boltz-2 and AlphaFold 2 agree with the crystal (mean TM 0.88 and 0.89) while ESMFold does not (0.22). Neither a stricter TM threshold nor an ESMFold confidence gate resolves the disagreement. MSA models recover a population-level advantage of natural structured junctions over a glycine-serine linker, but do not agree on which pairs benefit. Predictor scores are therefore not interchangeable on artificial fusions; the claim concerns the predictors, not the proteins.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
protein structure prediction, protein language models, chimeric fusion proteins

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_QSLGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines disagreement between ESMFold and MSA-based structure predictors on artificial two-domain fusion proteins. The authors construct 1,000 chimeric domain pairs and evaluate preservation of the constituent folds against AFDB reference conformations. Comparisons with Boltz-2, AF2, and additional controls show that ESMFold produces more low-scoring predictions and exhibits domain-order preferences that do not transfer consistently across predictors. Further analyses distinguish isolated-domain prediction difficulty from fusion-dependent losses, examine pair-representation changes, and introduce a dummy-fusion screen for identifying context-sensitive domains.

Critical strengths
  1. A relevant evaluation question supported by extensive controls. The reliability of structure predictors as computational screening tools is important for protein design. The authors examine multiple predictors, alternative MSA procedures, linker variations, domain boundaries, and available experimental references. The independent redraw and clustered uncertainty estimates strengthen the reproducibility of the observed disagreement.

  2. An informative separation of failure regimes. Distinguishing domains that ESMFold already predicts poorly in isolation from domains affected specifically by fusion context improves the analysis beyond aggregate model comparisons. Recurrence across unrelated partners provides additional evidence that some context sensitivity is associated with particular domains.

Critical weaknesses
  1. The demonstrated contribution is too limited to establish a substantial advance in protein-design evaluation. The paper convincingly documents disagreement among predictors, but does not establish which predictions are correct for the artificial fusion constructs. Most evaluations use AFDB conformations as references, and the additional MSA-based predictors share evolutionary inputs. The deposited-structure analysis supports recovery of known source-domain folds, but does not validate those folds within the artificial fusions. Consequently, neither lower AFDB-referenced failure rates nor stronger agreement among MSA models establishes a more reliable design-screening procedure. The authors appropriately acknowledge this distinction; nevertheless, it leaves the operational recommendations without direct validation. The principal result remains a predictor-disagreement audit in a restricted construction setting, with insufficient evidence that it changes experimentally grounded design decisions.

  2. The extensions do not sufficiently overcome the descriptive nature of the study, and reproducibility is incomplete. Pair-representation drift localizes a correlated change but does not establish the mechanism responsible for context-dependent prediction loss. The dummy-fusion screen predicts a model-defined failure event, is evaluated using a single dummy partner, and reports perfect precision on a small, conditional subset. Its robustness and broader utility therefore remain uncertain. Moreover, the supplementary archive explicitly omits analysis scripts. Since the study relies on custom construction, mapping, scoring, statistical resampling, and representation extraction, result tables alone are insufficient for transparent reproduction or reuse. Taken together, these limitations leave neither a sufficiently developed diagnostic method nor a fully reproducible benchmark contribution.

Questions for authors
  1. Practical significance. Beyond documenting disagreement, what validated design decision can be improved using the findings? In particular, what supports the recommendation involving joint MSA-model rejection beyond its lower rejection rate?

  2. Reference dependence. How should readers distinguish preservation of an AFDB source-domain conformation from structural correctness in the artificial fusion? Please clarify which conclusions remain supported without treating either the reference conformation or predictor consensus as fusion-level ground truth.

  3. Unseen-combination claims. Were assembled sequences or family combinations checked against broader UniProt collections, or only against the natural source pool? Please align claims about absence from UniProt and training-distribution novelty with the checks actually performed.

  4. Diagnostic validation. How sensitive is the dummy-fusion screen to the selected partner? Please clarify the population used for its precision and recall estimates and distinguish prospective screening performance from performance conditional on an already identified predictor-disagreement subset.

  5. Reproducibility. Can the authors provide the construction and analysis scripts, exact inference configurations, and commands required to reproduce the main results, including domain mappings, TM-score calculations, clustered resampling, and representation-drift measurements?

Reviewer_MJHJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the reliability and agreement of structure predictors on artificial multi-domain fusion proteins, comparing ESMFold with the MSA-based predictors, Boltz-2 and AlphaFold 2. The authors construct a benchmark of 1,000 chimeric domain pairs from different Swiss-Prot source proteins, evaluate both domain orders, and use family-disjoint folds for supervised analyses.

ESMFold exhibits domain-order preferences that are reproducible across alternative linkers (r=0.90r = 0.90) but poorly correlated with Boltz-2’s preferences (r=0.07r = 0.07). Selecting the order preferred by ESMFold does not clearly improve scores under the other predictors. On a randomly sampled panel of 300 constructs, ESMFold assigns low quality (q<0.7q < 0.7) to 17.7% of constructs, compared with 8.7% for Boltz-2 and 4.0% for AF2. AF2 largely agrees with Boltz-2 on ESMFold-only failures, and available crystal references support this pattern for six affected domains.

ESMFold-only failures divide into difficulties already present in isolated domains and failures arising after fusion. The latter are associated with changes in internal pair representations, without establishing causality. Neither the tested TM-score thresholds nor confidence gates resolve the disagreement. The findings therefore caution against relying on ESMFold alone to filter fusion designs, while leaving the actual folding and function of these artificial proteins experimentally unresolved.

Critical strengths

The topic of the paper is of great relevance. The paper examines the established concern of disagreement among protein structure predictors in the specific setting of chimeric fusion proteins. The authors introduce a set of controlled benchmarks of 1,000 domain pairs, on chimeric sequences taken from various domains and linkers while keeping domain sequences fixed. Isolated-domain controls, a second independently sampled panel, and comparisons with additional predictors strengthen the evidence for reproducible disagreement. The experimental setup and the dataset construction are detailed. In the experiments in particular, there is great attention in appropriately distinguishing between predictor behavior and experimentally established protein folding or function.

Critical weaknesses

The introduced benchmark provides only 1,000 domain pairs, although the main three-predictor comparison covers only 150 pairs, corresponding to 300 constructs, with a second disjoint panel providing replication. This limits the breadth of the cross-model evaluation. The evaluation primarily relies on AFDB-predicted reference structures, potentially favoring AlphaFold-related predictions. Experimental references partially address this concern but cover only a small subset and do not validate the artificial fusions themselves.

The worse-domain TM-score assesses preservation of individual domain conformations but does not evaluate inter-domain arrangement, stability, or biological function. Agreement among MSA-based predictors therefore cannot establish which predictions accurately describe the complete fusion proteins The figures and plots supporting the current finding are not actually explicative enough, lacking clarity in both the method application and experimental construction. The association between internal pair-representation drift and fusion-context failures does not establish a causal mechanism.

Questions for authors

a. Is the observed disagreement a phenomenon that also occurs in natural multi-domain proteins or other engineered constructs? Could analyzing additional cases help establish which findings are specific to the considered setting?

b. What computational or data constraints motivated restricting the main three-predictor comparison to 150 domain pairs? Would expanding this comparison alter the estimated failure rates or reveal differences across domain families?

c. How dependent are the conclusions on the selected model versions and inference settings? Could broader comparisons with additional single-sequence and MSA-based predictors clarify these effects?

d. Given that the MSA-based predictors share evolutionary inputs and the main references originate from AFDB, how do you distinguish genuine structural accuracy from correlated model biases?

e. Could multiple structural alignments of homologous domains provide a complementary evaluation reference, capturing conformational variability beyond a single AFDB structure?

Reviewer_M8vrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Paper is very clearly entirely AI written. The writing is so bad I could not read it.

Critical strengths

I cannot assess strengths as the paper is unreadable.

Critical weaknesses

Paper is AI written.

Questions for authors

To the extent that there is scientific merit to the work performed here, it is entirely obfuscated by the AI writing. Have a human rewrite from scratch.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.