- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether protein language models predict zero-shot mutation effects using constraints shared across evolutionary contexts or constraints specific to the target. Using synthetic Potts fitness landscapes with known mutation effects, the authors separate shared and target-specific contributions and intervene on training diversity and shared structure. Model scores align primarily with shared constraints, and suppressing those constraints substantially weakens the predictive signal with little compensation from target-specific structure. Analyses of ESM2 on ProteinGym show a similar pattern and find that homolog-cluster identity is decodable from hidden states, with limited evidence that it guides mutation scoring.
Critical strengths
1. The interventions strengthen the evidence that shared constraints drive prediction.
At a fixed budget of 5,120 sequences, increasing training landscapes from 4 to 32 improves shared alignment across all three model scales, for example from 0.054 to 0.159 for the smallest model. Conversely, suppressing shared structure while preserving parameter RMS reduces shared alignment to 0.012–0.028, with little compensation from target-specific effects. These complementary interventions provide evidence beyond the correlational decomposition and are a major reason I lean toward acceptance.
2. The paper directly examines whether decoded context is reflected in predictions.
Support-sequence classification accuracy reaches 96.3–98.0%, but remote-query classification remains near chance, and incorrect assignments do not consistently predict the corresponding mutation profiles. The natural-protein analysis likewise finds a small attribution-aligned margin of 0.007, with a 95% confidence interval of [−0.008, 0.022]. Connecting probe assignments to mutation predictions provides a useful check on what decodable context actually tells us about model behavior.
Critical weaknesses
1. Low predictive accuracy leaves incomplete learning as a plausible explanation.
Synthetic target correlations are only 0.107–0.137, so the reported 80–85% shared contribution concerns a relatively weak predictive signal. The models may simply learn broadly supported constraints before harder target-specific effects. Agreement across three model sizes does not resolve this concern. Data-scaling and convergence checks, or an explicitly context-conditioned control, would help establish whether shared dominance persists when the task is learned more effectively.
2. The central findings need replication across independent synthetic worlds.
The confidence intervals resample 16 landscapes within one generated world, leaving variation across worlds and training seeds unquantified. Although some auxiliary experiments use multiple worlds, the central interventions lack comparable replication. All landscapes also share a fixed interaction topology, which may influence the result. Repeating the main experiments across independent worlds and seeds, with a sensitivity check allowing topology changes, would help determine how broadly the conclusions hold.
Questions for authors
1. Replication of the central results across worlds and training seeds.
The intervals in Figure 1 resample the 16 held-out landscapes within one generated world and do not capture variation across worlds or training seeds. Several auxiliary experiments include independent worlds, which makes the absence of comparable replication for the central results consequential.
Could the authors clarify whether independent replications of the Figure 1 experiments are available but unreported? If so, please report the shared-versus-specific decomposition, the K-sweep increments, and the suppression contrasts separately by world and training seed.
If such results are unavailable, I request additional replication of these central experiments across independently generated worlds and multiple seeds per world. Reporting world-level effects, with training-seed variability distinguished from world-to-world variability, would directly address this concern.
This is the main uncertainty affecting my assessment. Consistent effects across independent worlds would substantially strengthen my confidence and could raise my score. Large variation or reversals would limit the generality of the central finding.
2. Does successful context identification predict stronger target-specific alignment?
The same-wrong profile margin examines incorrect context assignments, while local remote-query classification accuracy is only 0.344–0.383, close to the one-third chance level. This makes it difficult to separate unreliable context retrieval from failure to use successfully retrieved context.
Using the existing representations and mutation predictions, could the authors compare target-specific residual alignment, or its incremental rank-R² contribution, between correct and incorrect local context assignments? Comparisons within landscapes and with sequence identity controlled would help reduce confounding. Please also report the number of observations and uncertainty for each subset.
This analysis would affect how I interpret the representation–prediction result. Stronger specific alignment under correct assignments would suggest that context retrieval contributes to prediction and that retrieval failure explains part of the remote-query behavior. Little difference, with adequate statistical precision, would strengthen the evidence that decodable context and target-specific prediction remain separate in this setting.
2 comments
Response to Reviewer pH7P (continued)
Fixed interaction topology
We generated additional worlds with edge-rewiring probabilities of 0.05 and 0.10 per landscape-tree edge, which reduce mean pair overlap to 0.320 and 0.179. Target–shared oracle similarity falls to 0.611 and 0.481, and exact-target shared fraction to 0.301 and 0.158. Across both rewiring levels, two seeds, and three scales, exceeds in shared alignment in 12/12 configurations, suppression decreases shared alignment in 12/12, and PLM shared fraction exceeds the exact-target reference in 12/12. Appendix A.3 gives the numerical rows rather than sign counts alone.
Does successful context identification predict greater target-specific alignment?
We reran this analysis on the same optimization-matched checkpoints used for Figure 1 across three worlds, two training seeds, and three scales (18 models), rebuilding the remote same-landscape cohorts and identity-matched candidate triplets with the same outcome-blind rules in each world. Each world contains 256 remote queries; the strict subsets retain 156, 151, and 134 queries. For support and query accuracies and same-wrong margins, we report the range across the three seed-averaged worlds. For correct-versus-incorrect effects, we average seeds within each held-out landscape and bootstrap the resulting 48 landscapes (50,000 replicates; fixed seed).
Support contexts remain separable, but local remote-query accuracy is only 0.345/0.361/0.351 for xs/l/xxxl (chance = 1/3), and 0.338/0.347/0.347 under strict matching. Whole-sequence same-wrong margins are small and positive (0.0064/0.0100/0.0063; world ranges 0.0030–0.0111, 0.0075–0.0127, and 0.0030–0.0086), consistent with a weak distributed context effect. Masked-site same-wrong margins are // for xs/l/xxxl, with the seed-averaged world ranges spanning zero.
Correct retrieval does not increase target-specific alignment. In the full cohorts, identity-adjusted correct-minus-incorrect PLM–specific correlation is , , and for xs/l/xxxl. In the strict subsets the differences are , , and . Incremental specific rank- likewise shows no positive pooled effect. The positive strict-xxxl association in the submitted reference-world checkpoint does not replicate in this matched 18-model analysis, so we have removed the interpretation that successful retrieval contributes specifically at the largest scale.
Summary
Each analysis requested by the reviewer is now in the revision. The central interventions are replicated in three independently generated worlds with two seeds, with results reported by world and seed, and in worlds with rewired interaction topology. The intervention directions are consistent in all 18 fixed-topology configurations and all 12 rewired configurations. Incomplete learning does not explain the target-specific deficit. The deficit persists at data in all three worlds, where target Spearman reaches 0.50–0.67, and the same architectures recover the residual when trained on one landscape. The correct-versus-incorrect analysis shows no association between retrieval and target-specific alignment. Two results are narrower than in the original submission and are stated as such in the revised paper. Namely, the breadth effect is mainly first-order, and the excess over the exact-target reference is not consistent for the xxxl model at the base budget. The central conclusion does not depend on either.
Response to Reviewer pH7P
We thank the reviewer for identifying incomplete learning and replication across worlds and seeds as the main uncertainties. We added direct controls for both, and the requested correct-versus-incorrect context analysis. (Please note that terms are defined in the summary of revisions above.)
Incomplete-learning alternative
Two results argue against incomplete learning as the explanation for the target-specific deficit.
First, the same architectures recover the target-specific structure when trained on a single landscape at the same 5,120-sequence budget.On the same eight landscapes and query sets, specialist-minus-pooled incremental specific rank-R2 is +0.065 [0.024,0.109], +0.039 [0.016,0.064], and +0.049 [0.017,0.084] on near queries, and +0.058 [0.035,0.080], +0.044 [0.031,0.057], and +0.045 [0.031,0.060] on remote queries for the xs/l/xxxl models. This is therefore not only a raw-correlation effect.
Second, we scaled heterogeneous K=32 training from 5,120 to 10,240 and 20,480 sequences. At 2× data, target Spearman is 0.372/0.235/0.145 and incremental specific rank-R2 remains 0.0072/0.0072/0.0050. At 4× in the reference world, target Spearman reaches 0.673/0.668/0.636, incremental shared rank-R2 rises to 0.516/0.502/0.472, and incremental specific rank-R2 remains 0.0042/0.0045/0.0047. We then repeated 4× K=32 training in the two additional fixed-topology worlds with the same model seed. All six added models pass validation selection. Target Spearman is 0.495–0.571. Shared fraction is 0.974–0.985. Incremental specific rank-R2 remains 0.0051–0.0085. Across all three worlds and three scales, the nine 4× models therefore remain shared-dominant as overall learning improves. The additional data come from independent sequence-tree sets per training landscape, not additional descendants of the original trees. Sequence hashes confirm zero overlap with validation or query cohorts. Validation-loss trajectories for the reference-world scaling series and selected epochs for all 4× replications are now reported in the expanded Appendix A.3.
The independent-site analysis gives the same conclusion at the 20,480-sequence budget. At 20,480 sequences, controlling for an empirical per-site profile still leaves PLM–shared partial correlations of 0.703/0.703/0.669 and incremental shared rank-R2 beyond the profile of 0.421/0.442/0.400. The 4× improvement is therefore not merely better estimation of marginal residue frequencies.
The breadth intervention itself is more limited after this control. At the 5,120-sequence budget, incremental shared rank-R2 beyond the profile does not increase with K. We therefore trained K=4 at the same 20,480-sequence budget as K=32: the profile-controlled K=32 minus K=4 partial-correlation advantage is +0.026 [0.005,0.047] for the xs model, +0.008 [−0.009,0.027] for the l model, and −0.0002 [−0.0226,0.0186] for the xxxl model, while incremental shared rank-R2 beyond the profile does not differ reliably at any scale. Breadth thus increases raw shared alignment, but we do not claim that breadth alone consistently increases the higher-order incremental shared component.
Replication across worlds and training seeds
We repeated the optimization-matched K=4/K=32 and suppression experiments in three independent fixed-topology worlds with two training initializations per world. Appendix A.3 now reports one row for every world, seed, and scale, including each world's exact-target shared fraction.
The K=32 minus K=4 shared contrast is positive in all 18 world–seed–scale configurations, and suppression decreases PLM–shared alignment in all 18. Averaged over worlds and seeds, the breadth effect is +0.101/+0.078/+0.066 for the xs/l/xxxl models, while suppression changes shared alignment by −0.123/−0.108/−0.095. The target-specific suppression response is mixed (8 positive, 10 negative) and averages approximately zero (+0.0003/+0.0010/−0.0034).
The exact-target shared fraction varies across worlds: 0.775, 0.716, and 0.702. PLM shared fraction exceeds its own world's exact-target fraction in 15/18 fixed-topology configurations. The excess is clear for the xs model (6/6 configurations; mean +0.096), smaller for the l model (5/6; +0.069), and not consistent for the xxxl model at the base budget (4/6; +0.027). The intervention directions nevertheless do not reverse.
To separate the two sources of variability, we report the SD across worlds and the pooled within-world SD across seeds. For the breadth effect these are 0.034/0.042, 0.016/0.011, and 0.027/0.006 for the xs/l/xxxl models, respectively. For suppression they are 0.020/0.046, 0.030/0.022, and 0.031/0.010 respectively.