acceptodds
Under review as a conference paper at ICLR 2027

NOT A FITNESS BOTTLENECK: WHY PROTEIN LANGUAGE MODELS FALL SHORT ON DISORDERED PROTEIN ENSEMBLES

Abstract

Protein language models (PLMs) are increasingly used as frozen encoders to predict the conformational properties of intrinsically disordered proteins (IDPs), yet how much ensemble physics evolutionary pretraining captures remains unclear. Here we measure evolutionary sufficiency (ES), the fraction of a directly supervised model's out-of-distribution accuracy that a frozen PLM representation recovers, for five polymer-physics properties computed from simulated ensembles of nearly 10,000 disordered sequences across six kingdoms, with viral sequences held out. We first show that ES is graded rather than binary, as frozen PLMs adapted to disordered regions recover nearly all of the supervised accuracy for chain dimensions but progressively less for asphericity, the Flory exponent, and its prefactor. We then test and reject a fitness-bottleneck explanation for this shortfall, because the embeddings predict these properties several times better than the PLM's own likelihood and encode net charge almost perfectly, whereas mean pooling hides the charge patterning on which the hardest properties depend. Instead, the gap has a readout component, which a position-aware probe partly closes and which grows with model size, and a pretraining-data component, as encoders pretrained on simulated ensembles reach the supervised ceiling while frozen. Once the backbone is fine-tuned, however, evolutionary initialisation matters little. ES thus offers a probe-aware diagnostic for when frozen PLM features suffice for biophysical prediction.

Primary area:
learning theory
Keywords:
protein language models, intrinsically disordered proteins, information theory, evolutionary fitness, out-of-distribution generalization, representation learning, molecular dynamics, inverse reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EYwNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper inverstiagyes if Protein Langauge Models (PLMs) encode useful dynamics information of IDPs. They measure the fraction of performence recovered using PLM embeddings, models trained with physical objectives.

Critical strengths

The comparison is used to justify a rejection for the fitness botelneck hypothesis. Directly testing weather the embeddings contain information that the estimated fitness function doesn't.

Critical weaknesses

An unaddressed question is whether this finding is specific to IDPs or holds for folded proteins too. If it holds generally, it says something broad about PLMs. If not, it’s a narrower result about disordered sequences.

Questions for authors

See weaknesses

Reviewer_3vUrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary: The paper proposed a method to measure how much information protein language models (PLMs) encode about the properties of IDPs. The authors introduce evolutionary sufficiency (ES) that is the fraction of the out-of-distribution performance of a supervised model that a frozen PLM representation recovers and they measure it for 5 properties.

Contributions:

  1. They give a way to measure how much a frozen pLM suffices for biophysical IDP prediction.
  2. They test and reject the hypothesis that pLM embeddings can only recover properties related to fitness.
  3. They explain the decrease in ES into readout and pretraining-data components.
Critical strengths
  1. Carefully designed test that fills a real gap: The authors propose a causal explanation (the fitness bottleneck) and design a controlled experiment to test it. This is valuable for the community since the field lacks principled tools for understanding why pLMs succeed or fail on IDPs.

  2. The fitness hypothesis is rejected (using 3 approaches) and replaced with an alternative explanation: All three predictions of the fitness bottleneck are rejected with quantitative evidence and the authors explain the decrease in ES using the readout and pretraining-data components.

Critical weaknesses
  1. The authors use the log p as a fitness proxy based on the claim that "Masked language modelling on extant sequences can be viewed as maximum-entropy inverse reinforcement learning on the evolutionary fitness landscape (Jiao et al., 2026; Ziebart et al., 2008)". But this is a theoretical claim, as far as I understand. How can we be sure that the log p of the pLMs used give accurate estimations of the actual fitness?

  2. The study is tied to a single simulation framework from CALVADOS-2 trajectories. It is therefore unclear whether the conclusions would hold for labels derived from all-atom simulations or other experimental measurements.

Questions for authors
  1. Can you provide any empirical validation that log p correlates with actual fitness measurements for the sequences in your dataset?
  2. All targets are derived from CALVADOS-2 trajectories. Can you validate the framework using an additional simulation framework or experimental measurements to assess whether the conclusions generalise beyond CALVADOS-2?

minor: a reference is missing in line 129 "zero-shot ()"

Reviewer_sjemOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how well frozen protein language models (PLMs) capture the conformational properties of intrinsically disordered proteins (IDPs). The authors introduce evolutionary sufficiency (ES), measuring the fraction of supervised prediction accuracy recovered by frozen representations. Through experiments on five physical properties, they conclude that PLMs capture chain dimensions well but struggle with more complex properties involving sequence patterning. They further argue that these limitations arise from readout and pretraining data rather than an evolutionary fitness bottleneck.

Critical strengths

Comprehensive empirical analysis. The paper evaluates multiple PLMs, model sizes, probing architectures, and training strategies, including random initialization and supervised baselines. The experimental comparisons are generally well controlled.

Critical weaknesses

1 The rejection of the fitness-bottleneck hypothesis is not sufficiently justified. The hypothesis in Equation (2) assumes conditional independence between the representation (T) and target (Y) given evolutionary fitness (F). However, the authors' prediction P3 does not follow from this assumption. For example, consider (T=F) and (Y=F^2), where (F) is uniformly distributed over ([-1,1]). The conditional-independence hypothesis holds exactly, yet a linear probe cannot predict (Y) better than a constant predictor, while a nonlinear probe can predict it perfectly. Therefore, improved performance from deeper probes does not contradict the fitness-bottleneck hypothesis.

2 The empirical observations are largely unsurprising, limiting the novelty of the contribution. The main findings are that larger or more expressive probes can improve predictions from frozen PLM embeddings, and that models pretrained directly on simulated physical properties outperform models pretrained only on evolutionary sequences. In particular, GeoGraph was already pretrained to predict the same five properties using simulations from the same force-field family as the evaluation data. These observations are largely expected, and the paper does not provide sufficiently new methodological or conceptual insights beyond these findings.

Questions for authors

1 Fitness-bottleneck hypothesis (W1). Could the authors clarify why P3 follows from Equation (2)? Also, how can they distinguish a failure of the hypothesis from an inaccurate fitness proxy?

2 Readout architecture. The deep probe has ~ 56x more parameters than the shallow probe and changes both pooling and model capacity. Could the authors compare attention pooling against mean pooling with matched probe capacities to isolate the source of improvement?

3 Simulator pretraining. Since GeoGraph was pretrained to predict the same properties using the same force-field family, could the authors evaluate transfer to labels generated by an independent simulator or experimental measurements?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.