acceptodds
Under review as a conference paper at ICLR 2027

Shared Constraints Dominate Protein Language Model Generalization

Abstract

When foundation models are trained on data pooled across heterogeneous generative contexts, successful zero-shot transfer can occur via two distinct mechanisms. The model may dynamically condition on latent context-specific structure, or it may rely primarily on regularities invariant across the training distribution. Protein language models (PLMs) provide an opportune testbed for this distinction, as they are trained on diverse evolutionary corpora and yet evaluated on zero-shot mutation-effect prediction defined by individual, context-dependent fitness landscapes. In this paper, using a controlled synthetic evolutionary framework with exact oracles and then a natural-corpora setting, we investigate whether PLMs learn task-invariant or task-specific structure. In both settings we find that PLM predictions track mutation effects common across training contexts rather than specific to a target. Intervening on training diversity reveals that broader ensembles strengthen this shared signal, while suppressing shared structure causes prediction to collapse rather than triggering a switch to task-specific structure. The synthetic setting reveals that learned representations encode fine-grained information about which lineage a sequence belongs to. Similarly, in the natural setting, where lineage is operationalized as a cluster of similar homologs, cluster identity is indeed decodable from local hidden states. However, we find that models of various scales do not employ this information to predict mutation effects. Altogether, our findings suggest that PLM generalization on mutation tasks is fundamentally driven by shared constraints, providing new insights into how sequence models may handle latent distribution shifts.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
protein language models, evolutionary generalization, constraint integration, mutation-effect prediction, pretraining data composition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_pH7POpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether protein language models predict zero-shot mutation effects using constraints shared across evolutionary contexts or constraints specific to the target. Using synthetic Potts fitness landscapes with known mutation effects, the authors separate shared and target-specific contributions and intervene on training diversity and shared structure. Model scores align primarily with shared constraints, and suppressing those constraints substantially weakens the predictive signal with little compensation from target-specific structure. Analyses of ESM2 on ProteinGym show a similar pattern and find that homolog-cluster identity is decodable from hidden states, with limited evidence that it guides mutation scoring.

Critical strengths

1. The interventions strengthen the evidence that shared constraints drive prediction.

At a fixed budget of 5,120 sequences, increasing training landscapes from 4 to 32 improves shared alignment across all three model scales, for example from 0.054 to 0.159 for the smallest model. Conversely, suppressing shared structure while preserving parameter RMS reduces shared alignment to 0.012–0.028, with little compensation from target-specific effects. These complementary interventions provide evidence beyond the correlational decomposition and are a major reason I lean toward acceptance.

2. The paper directly examines whether decoded context is reflected in predictions.

Support-sequence classification accuracy reaches 96.3–98.0%, but remote-query classification remains near chance, and incorrect assignments do not consistently predict the corresponding mutation profiles. The natural-protein analysis likewise finds a small attribution-aligned margin of 0.007, with a 95% confidence interval of [−0.008, 0.022]. Connecting probe assignments to mutation predictions provides a useful check on what decodable context actually tells us about model behavior.

Critical weaknesses

1. Low predictive accuracy leaves incomplete learning as a plausible explanation.

Synthetic target correlations are only 0.107–0.137, so the reported 80–85% shared contribution concerns a relatively weak predictive signal. The models may simply learn broadly supported constraints before harder target-specific effects. Agreement across three model sizes does not resolve this concern. Data-scaling and convergence checks, or an explicitly context-conditioned control, would help establish whether shared dominance persists when the task is learned more effectively.

2. The central findings need replication across independent synthetic worlds.

The confidence intervals resample 16 landscapes within one generated world, leaving variation across worlds and training seeds unquantified. Although some auxiliary experiments use multiple worlds, the central interventions lack comparable replication. All landscapes also share a fixed interaction topology, which may influence the result. Repeating the main experiments across independent worlds and seeds, with a sensitivity check allowing topology changes, would help determine how broadly the conclusions hold.

Questions for authors

1. Replication of the central results across worlds and training seeds.

The intervals in Figure 1 resample the 16 held-out landscapes within one generated world and do not capture variation across worlds or training seeds. Several auxiliary experiments include independent worlds, which makes the absence of comparable replication for the central results consequential.

Could the authors clarify whether independent replications of the Figure 1 experiments are available but unreported? If so, please report the shared-versus-specific decomposition, the K-sweep increments, and the suppression contrasts separately by world and training seed.

If such results are unavailable, I request additional replication of these central experiments across independently generated worlds and multiple seeds per world. Reporting world-level effects, with training-seed variability distinguished from world-to-world variability, would directly address this concern.

This is the main uncertainty affecting my assessment. Consistent effects across independent worlds would substantially strengthen my confidence and could raise my score. Large variation or reversals would limit the generality of the central finding.

2. Does successful context identification predict stronger target-specific alignment?

The same-wrong profile margin examines incorrect context assignments, while local remote-query classification accuracy is only 0.344–0.383, close to the one-third chance level. This makes it difficult to separate unreliable context retrieval from failure to use successfully retrieved context.

Using the existing representations and mutation predictions, could the authors compare target-specific residual alignment, or its incremental rank-R² contribution, between correct and incorrect local context assignments? Comparisons within landscapes and with sequence identity controlled would help reduce confounding. Please also report the number of observations and uncertainty for each subset.

This analysis would affect how I interpret the representation–prediction result. Stronger specific alignment under correct assignments would suggest that context retrieval contributes to prediction and that retrieval failure explains part of the remote-query behavior. Little difference, with adequate statistical precision, would strengthen the evidence that decodable context and target-specific prediction remain separate in this setting.

2 comments
AuthorsOpenReview ↗

Response to Reviewer pH7P

We thank the reviewer for identifying incomplete learning and replication across worlds and seeds as the main uncertainties. We added direct controls for both, and the requested correct-versus-incorrect context analysis. (Please note that terms are defined in the summary of revisions above.)

Incomplete-learning alternative

Two results argue against incomplete learning as the explanation for the target-specific deficit.

First, the same architectures recover the target-specific structure when trained on a single landscape at the same 5,120-sequence budget.On the same eight landscapes and query sets, specialist-minus-pooled incremental specific rank-R2R^2 is +0.065+0.065 [0.024,0.109][0.024, 0.109], +0.039+0.039 [0.016,0.064][0.016, 0.064], and +0.049+0.049 [0.017,0.084][0.017, 0.084] on near queries, and +0.058+0.058 [0.035,0.080][0.035, 0.080], +0.044+0.044 [0.031,0.057][0.031, 0.057], and +0.045+0.045 [0.031,0.060][0.031, 0.060] on remote queries for the xs/l/xxxl models. This is therefore not only a raw-correlation effect.

Second, we scaled heterogeneous K=32K=32 training from 5,120 to 10,240 and 20,480 sequences. At 2×2\times data, target Spearman is 0.372/0.235/0.145 and incremental specific rank-R2R^2 remains 0.0072/0.0072/0.0050. At 4×4\times in the reference world, target Spearman reaches 0.673/0.668/0.636, incremental shared rank-R2R^2 rises to 0.516/0.502/0.472, and incremental specific rank-R2R^2 remains 0.0042/0.0045/0.0047. We then repeated 4×4\times K=32K=32 training in the two additional fixed-topology worlds with the same model seed. All six added models pass validation selection. Target Spearman is 0.495–0.571. Shared fraction is 0.974–0.985. Incremental specific rank-R2R^2 remains 0.0051–0.0085. Across all three worlds and three scales, the nine 4×4\times models therefore remain shared-dominant as overall learning improves. The additional data come from independent sequence-tree sets per training landscape, not additional descendants of the original trees. Sequence hashes confirm zero overlap with validation or query cohorts. Validation-loss trajectories for the reference-world scaling series and selected epochs for all 4×4\times replications are now reported in the expanded Appendix A.3.

The independent-site analysis gives the same conclusion at the 20,480-sequence budget. At 20,480 sequences, controlling for an empirical per-site profile still leaves PLM–shared partial correlations of 0.703/0.703/0.669 and incremental shared rank-R2R^2 beyond the profile of 0.421/0.442/0.400. The 4×4\times improvement is therefore not merely better estimation of marginal residue frequencies.

The breadth intervention itself is more limited after this control. At the 5,120-sequence budget, incremental shared rank-R2R^2 beyond the profile does not increase with KK. We therefore trained K=4K=4 at the same 20,480-sequence budget as K=32K=32: the profile-controlled K=32K=32 minus K=4K=4 partial-correlation advantage is +0.026+0.026 [0.005,0.047][0.005, 0.047] for the xs model, +0.008+0.008 [−0.009,0.027][-0.009, 0.027] for the l model, and −0.0002-0.0002 [−0.0226,0.0186][-0.0226, 0.0186] for the xxxl model, while incremental shared rank-R2R^2 beyond the profile does not differ reliably at any scale. Breadth thus increases raw shared alignment, but we do not claim that breadth alone consistently increases the higher-order incremental shared component.

Replication across worlds and training seeds

We repeated the optimization-matched K=4K=4/K=32K=32 and suppression experiments in three independent fixed-topology worlds with two training initializations per world. Appendix A.3 now reports one row for every world, seed, and scale, including each world's exact-target shared fraction.

The K=32K=32 minus K=4K=4 shared contrast is positive in all 18 world–seed–scale configurations, and suppression decreases PLM–shared alignment in all 18. Averaged over worlds and seeds, the breadth effect is +0.101+0.101/+0.078+0.078/+0.066+0.066 for the xs/l/xxxl models, while suppression changes shared alignment by −0.123-0.123/−0.108-0.108/−0.095-0.095. The target-specific suppression response is mixed (8 positive, 10 negative) and averages approximately zero (+0.0003+0.0003/+0.0010+0.0010/−0.0034-0.0034).

The exact-target shared fraction varies across worlds: 0.775, 0.716, and 0.702. PLM shared fraction exceeds its own world's exact-target fraction in 15/18 fixed-topology configurations. The excess is clear for the xs model (6/6 configurations; mean +0.096+0.096), smaller for the l model (5/6; +0.069+0.069), and not consistent for the xxxl model at the base budget (4/6; +0.027+0.027). The intervention directions nevertheless do not reverse.

To separate the two sources of variability, we report the SD across worlds and the pooled within-world SD across seeds. For the breadth effect these are 0.034/0.042, 0.016/0.011, and 0.027/0.006 for the xs/l/xxxl models, respectively. For suppression they are 0.020/0.046, 0.030/0.022, and 0.031/0.010 respectively.

AuthorsOpenReview ↗

Response to Reviewer pH7P (continued)

Fixed interaction topology

We generated additional worlds with edge-rewiring probabilities of 0.05 and 0.10 per landscape-tree edge, which reduce mean pair overlap to 0.320 and 0.179. Target–shared oracle similarity falls to 0.611 and 0.481, and exact-target shared fraction to 0.301 and 0.158. Across both rewiring levels, two seeds, and three scales, K=32K=32 exceeds K=4K=4 in shared alignment in 12/12 configurations, suppression decreases shared alignment in 12/12, and PLM shared fraction exceeds the exact-target reference in 12/12. Appendix A.3 gives the numerical rows rather than sign counts alone.

Does successful context identification predict greater target-specific alignment?

We reran this analysis on the same optimization-matched K=32K=32 checkpoints used for Figure 1 across three worlds, two training seeds, and three scales (18 models), rebuilding the remote same-landscape cohorts and identity-matched candidate triplets with the same outcome-blind rules in each world. Each world contains 256 remote queries; the strict subsets retain 156, 151, and 134 queries. For support and query accuracies and same-wrong margins, we report the range across the three seed-averaged worlds. For correct-versus-incorrect effects, we average seeds within each held-out landscape and bootstrap the resulting 48 landscapes (50,000 replicates; fixed seed).

Support contexts remain separable, but local remote-query accuracy is only 0.345/0.361/0.351 for xs/l/xxxl (chance = 1/3), and 0.338/0.347/0.347 under strict matching. Whole-sequence same-wrong margins are small and positive (0.0064/0.0100/0.0063; world ranges 0.0030–0.0111, 0.0075–0.0127, and 0.0030–0.0086), consistent with a weak distributed context effect. Masked-site same-wrong margins are +0.0013+0.0013/+0.0002+0.0002/−0.0001-0.0001 for xs/l/xxxl, with the seed-averaged world ranges spanning zero.

Correct retrieval does not increase target-specific alignment. In the full cohorts, identity-adjusted correct-minus-incorrect PLM–specific correlation is +0.0006+0.0006 [−0.0039,0.0051][-0.0039, 0.0051], +0.0001+0.0001 [−0.0049,0.0051][-0.0049, 0.0051], and −0.0018-0.0018 [−0.0066,0.0029][-0.0066, 0.0029] for xs/l/xxxl. In the strict subsets the differences are +0.0005+0.0005 [−0.0072,0.0078][-0.0072, 0.0078], +0.0035+0.0035 [−0.0044,0.0114][-0.0044, 0.0114], and +0.0007+0.0007 [−0.0065,0.0080][-0.0065, 0.0080]. Incremental specific rank-R2R^2 likewise shows no positive pooled effect. The positive strict-xxxl association in the submitted reference-world checkpoint does not replicate in this matched 18-model analysis, so we have removed the interpretation that successful retrieval contributes specifically at the largest scale.

Summary

Each analysis requested by the reviewer is now in the revision. The central interventions are replicated in three independently generated worlds with two seeds, with results reported by world and seed, and in worlds with rewired interaction topology. The intervention directions are consistent in all 18 fixed-topology configurations and all 12 rewired configurations. Incomplete learning does not explain the target-specific deficit. The deficit persists at 4×4\times data in all three worlds, where target Spearman reaches 0.50–0.67, and the same architectures recover the residual when trained on one landscape. The correct-versus-incorrect analysis shows no association between retrieval and target-specific alignment. Two results are narrower than in the original submission and are stated as such in the revised paper. Namely, the breadth effect is mainly first-order, and the excess over the exact-target reference is not consistent for the xxxl model at the base budget. The central conclusion does not depend on either.

Reviewer_q63XOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
1(−1)
Presentation
1(−1)
Soundness
1
Read review
Paper summary

The paper studies whether zero-shot mutation scores from protein language models reflect constraints shared across evolutionary contexts or constraints specific to a target. A hierarchical Potts simulation provides exact mutation-effect oracles, enabling a shared/specific decomposition and interventions on training diversity and shared structure. An identity-matched cluster-Potts analysis finds a similar ordering across ESM2 scales on natural proteins. Representation analyses test whether centroid-assigned contexts predict the mutation profiles expressed by the models.

Critical strengths
  1. The controlled framework separates sequence remoteness from a change in the governing fitness landscape. Exact oracles and the remote same-landscape control make this a useful contribution to evaluating mutation-effect generalization.
  2. The natural analysis has thoughtful safeguards: outcome-blind cohort selection, identity matching, split-half reliability estimates, and an errors-in-variables sensitivity. Agreement with DMS and consistency across the primary and sensitivity cohorts support the relevance of the observed ordering. The positive target-cluster advantage is also reported rather than omitted.
  3. The profile-level and permutation controls are more informative than scalar proximity to an alternative cluster. The appendices document the experiments extensively and distinguish the main decomposition from the larger natural breadth analyses. They also state several important limits, including what the single-world bootstrap intervals measure.
Critical weaknesses
  1. The main intervention comparisons need a consistent optimization control. In the supplied reproduction path, run_e3_constraint_integration_extension_task.py fixes the learning rate to 2e-4 for K=4/8/16 and shared suppression. The runbook expects the selected K=32 checkpoints to use 5e-5 for l and xxxl. The final reporting script imports those selected K=32 results; the calibration stage does not retrain them. Thus the comparisons for these two scales change optimization as well as the intended intervention. The learning-rate discrepancy does not apply to xs, and all conditions select epochs on validation data, so it does not invalidate every intervention result. Nevertheless, it leaves the larger-model endpoint and suppression contrasts unresolved. There is a separate methods discrepancy: the suppression runner transforms the full 128-landscape bank before selecting training, validation, and test subsets, whereas Section 3.3 introduces the mean through the training landscapes. A global transformation of a synthetic world is legitimate, but the averaging population and RMS-preservation claim should be specified consistently.

  2. Shared dominance needs a reference before it establishes preferential recovery. A predictor that reproduces the complete target landscape can still receive a larger shared than specific incremental R2 when the target itself is mostly shared. Here, shared-target oracle Spearman correlation is approximately 0.90, while PLM-target Spearman is 0.107-0.137 (Section 4.1). The higher PLM correlation with shared than with complete-target effects is additional evidence in the authors' favor; the variance argument does not explain it away. Still, applying the same decomposition to the exact target oracle would clarify whether the reported 80-85% shared fraction of summed incremental R2 contributions indicates a deficit in specificity beyond the target's own composition. An accuracy-matched noisy-target reference would be a useful sensitivity, provided its assumptions are stated.

  3. The abstract overstates what the representation tests establish. Under strict matching, local remote-query accuracies are 0.355, 0.309, and 0.336 against chance 1/3, despite high support-context classification accuracy (Table 1). This does not establish that the correct context is reliably accessible for those remote queries but then ignored. The same-wrong margin instead tests whether an external centroid assignment acts as a discrete selector of a fitness profile. The natural attribution-aligned margin, +0.007 with CI [-0.008, 0.022] across seven proteins, is informative about that mechanism but does not exclude soft or distributed conditioning. The central synthetic experiments use one generated world, and the primary natural decomposition retains five of 173 eligible targets, with nine in the sensitivity cohort. The reliability correction and larger appendix studies are valuable, but do not establish the same decomposition across a broad biological population.

Questions for authors
  1. What run settings and checkpoints produced Figure 1? If the documented learning-rate difference applies to the reported results, can a K=32 control for l/xxxl use the same training and initialization rule as the extension conditions? Please also clarify whether suppression is centered on the full bank or training subset, and report the corresponding manipulation checks.
  2. What shared/specific increments does the existing analysis assign to the exact target oracle on the same queries?
  3. Can the representation conclusion be restricted to the tested centroid-aligned discrete-selection mechanism? If a broader non-use claim is retained, a positive control demonstrating sensitivity to a known context-using predictor would help.
  4. Section 4.4 compares a historical-weighted mixture of T, A, and B against an equal mixture of A and B. How much of the gain remains against equal-weight T/A/B?
  5. The framing would also benefit from discussing Weinstein et al. (2022), which addresses why observational sequence-density estimation and fitness estimation differ.
3 comments
AuthorsOpenReview ↗

Response to Reviewer q63X

We thank the reviewer for the code-level review. We agree that the learning-rate mismatch and the suppression-centering discrepancy had to be resolved before the larger-model intervention results could be interpreted. We reran the central conditions under matched optimization and made the manipulation definition and provenance explicit. (Please note that terms are defined in the summary of revisions above.)

Figure 1 provenance and optimization mismatch

The reviewer's reading of the submitted reproduction path is correct: the submitted K=32K=32 checkpoints for l and xxxl used learning rate 5×10−55 \times 10^{-5}, whereas the K=4/8/16K=4/8/16 extension and suppression path used 2×10−42 \times 10^{-4}. The revised Figure 1 does not mix those checkpoints. We retrained all K∈{4,8,16,32}K \in \{4, 8, 16, 32\} and suppression conditions under one scale-specific rule with paired initialization: learning rate 2×10−42 \times 10^{-4} for xs and 5×10−55 \times 10^{-5} for l/xxxl.

In the reference world, the K=32K=32 minus K=4K=4 PLM–shared Spearman contrast is +0.120+0.120 [0.082,0.163][0.082, 0.163], +0.083+0.083 [0.046,0.123][0.046, 0.123], and +0.088+0.088 [0.062,0.115][0.062, 0.115] for xs/l/xxxl. Across three independent fixed-topology worlds and two training seeds, the contrast is positive in all 18 world–seed–scale configurations. Appendix A.3 now reports every world/seed/scale row rather than only aggregate effects. The revised manuscript records the matched settings explicitly. We also updated the code supplement with a revision runbook that points to the matched Figure 1 runner, records the scale-specific learning rates, and documents training-centered suppression. The earlier mixed-optimization extension script is retained only to reproduce the originally submitted analysis and is not used for the revised Figure 1.

This raw KK effect needs qualification. At 5,120 sequences, empirical profile control leaves a positive K=32K=32 minus K=4K=4 partial-correlation contrast for xs and xxxl but not l, while incremental shared rank-R2R^2 beyond the profile does not increase with KK. We therefore also trained K=4K=4 at 20,480 sequences to match the existing K=32K=32 4×4\times condition. The profile-controlled partial-correlation advantage is +0.026+0.026 [0.005,0.047][0.005, 0.047] for xs, +0.008+0.008 [−0.009,0.027][-0.009, 0.027] for l, and −0.0002-0.0002 [−0.0226,0.0186][-0.0226, 0.0186] for xxxl. Incremental shared rank-R2R^2 beyond the profile again does not differ reliably at any scale. We therefore interpret breadth as increasing raw shared alignment but not as established evidence that breadth alone increases higher-order incremental shared recovery.

Suppression centering and manipulation checks

The revised intervention defines the common component from the 32 training landscapes and applies the transformation to all landscapes relative to that training mean, with residual scale calibrated to preserve pooled training parameter RMS exactly. This is not the same as preserving every landscape's RMS. In the reference world, the training common-variance fraction falls from 0.700 to 0.044 and mean per-landscape RMS changes from 0.959 to 0.947. On held-out landscapes the common-variance fraction falls from 0.742 to 0.125 and mean RMS from 0.958 to 0.921.

On the regenerated suppression evaluation cohort, target–shared oracle similarity falls from 0.898 to 0.324 and the exact-target shared fraction from 0.775 to 0.028. PLM–target Spearman correspondingly falls from 0.156/0.130/0.119 to 0.039/0.029/0.025 and PLM–shared Spearman from 0.182/0.155/0.137 to 0.049/0.037/0.022, while PLM–specific Spearman remains small at 0.032/0.024/0.022. Appendix A.3 now reports the manipulation checks and distinguishes the exactly preserved pooled training RMS from the changes in mean per-landscape RMS.

Exact-target and accuracy-matched references

Applying the identical drop-one decomposition to the exact target gives a shared fraction of 0.775 [0.745,0.806][0.745, 0.806]. The matched K=32K=32 PLM fractions are 0.886/0.861/0.837. The paired PLM-minus-target excess is +0.111+0.111 [0.072,0.141][0.072, 0.141], +0.086+0.086 [0.048,0.119][0.048, 0.119], and +0.062+0.062 [0.009,0.108][0.009, 0.108].

We also implemented the suggested accuracy-matched noisy-target sensitivity. Adding Gaussian noise until target Spearman matches each model yields shared fractions of 0.738 [0.712,0.762][0.712, 0.762], 0.726 [0.696,0.757][0.696, 0.757], and 0.720 [0.687,0.751][0.687, 0.751] over 1,000 draws, below the PLM fractions of 0.886/0.861/0.837. The same control was repeated for all 12 matched K=32K=32 checkpoints in the two additional fixed-topology worlds. PLM shared fraction exceeds the noisy-target reference in all 12, by 0.056–0.203. Low prediction accuracy therefore does not by itself explain the PLM's shared-dominant decomposition.

As a separate oracle sensitivity, kk-nearest training-landscape mixtures become better approximations of the held-out target but less correlated with PLM scores in all 180 nonuniform world–seed–scale comparisons. The uniform mean contributes more incremental PLM rank-R2R^2 in 178/180 joint comparisons (revised Appendix A.3).

AuthorsOpenReview ↗

Response to Reviewer q63X (continued)

Representation conclusion

We agree that the original abstract was too broad. The high support-context accuracy does not imply reliable retrieval on the remote scored queries. To place this analysis on the same checkpoints as Figure 1, we reran it on the optimization-matched K=32K=32 checkpoints for all three worlds, two seeds, and three scales (18 models), rebuilding the remote same-landscape cohorts and identity-matched candidate triplets in each world.

We find that support contexts remain separable, but local remote-query accuracy is only 0.345/0.361/0.351 for xs/l/xxxl (chance = 1/3), and 0.338/0.347/0.347 in the strict subsets. Whole-sequence assignments show a small positive same-wrong margin (means 0.0064/0.0100/0.0063. Seed-averaged world ranges 0.0030–0.0111, 0.0075–0.0127, and 0.0030–0.0086), which is compatible with weak distributed context use. The masked-site margins directly tied to the scored position are +0.0013+0.0013/+0.0002+0.0002/−0.0001-0.0001 for xs/l/xxxl, with seed-averaged world ranges [−0.0009,0.0028][-0.0009, 0.0028], [−0.0013,0.0028][-0.0013, 0.0028], and [−0.0012,0.0019][-0.0012, 0.0019].

For the correct-versus-incorrect endpoint, we average the two seeds within each held-out landscape and bootstrap the resulting 48 landscapes (50,000 replicates, fixed seed). Correct local retrieval does not predict greater target-specific alignment. In the full cohorts, identity-adjusted correct-minus-incorrect PLM–specific correlation is +0.0006+0.0006 [−0.0039,0.0051][-0.0039, 0.0051], +0.0001+0.0001 [−0.0049,0.0051][-0.0049, 0.0051], and −0.0018-0.0018 [−0.0066,0.0029][-0.0066, 0.0029] for xs/l/xxxl. In the strict subsets it is +0.0005+0.0005 [−0.0072,0.0078][-0.0072, 0.0078], +0.0035+0.0035 [−0.0044,0.0114][-0.0044, 0.0114], and +0.0007+0.0007 [−0.0065,0.0080][-0.0065, 0.0080]. Incremental specific rank-R2R^2 shows no positive pooled effect either. The earlier strict-xxxl positive result from the submitted reference-world checkpoint therefore does not replicate on the matched checkpoints, and we have removed the suggestion that retrieval contributes specifically at the largest scale.

We also agree that the five-target strict ProteinGym cohort does not establish this decomposition across a broad biological population. We therefore use the natural analysis only as concordant observational evidence, with the nine-target sensitivity and errors-in-variables correction as robustness checks.

Historical-weight baseline

We added the requested equal-T/A/B reference. Let M1=(A+B)/2M_1=(A+B)/2, M2=(T+A+B)/3M_2=(T+A+B)/3, and M3M_3 be the historically weighted T/A/B mixture. Adding the target cluster helps: M2−M1M_2 - M_1 improves mean ESM2 correlation by +0.031+0.031 to +0.048+0.048 across scales. Historical weighting does not improve over equal T/A/B: M3−M2M_3 - M_2 ranges from −0.0004-0.0004 to −0.0086-0.0086 across ESM2 scales (DMS: −0.0074-0.0074), and the true historical weight assignment is not systematically better than permutations. We have removed the claim that historical cluster proportions explain the gain (Appendix C.7).

Replication across worlds and topology

The matched central interventions were repeated in three independently generated worlds with two training seeds. The K=32K=32 minus K=4K=4 shared-alignment contrast is positive in 18/18 experimental configurations and suppression decreases shared alignment in 18/18. The exact-target composition itself varies across worlds (shared fractions 0.775/0.716/0.702). PLM shared fraction exceeds its own world's exact-target fraction in 15/18 configurations: the excess is clear for xs (6/6 configurations, mean +0.096+0.096), smaller for l (5/6; +0.069+0.069), and not consistent for xxxl at the base budget (4/6; +0.027+0.027). We therefore do not describe a uniform model-over-oracle excess at the largest scale at the base budget, while noting that the two intervention directions are consistent in sign in all configurations. At 4×4\times data, the shared-dominant decomposition replicates in the two additional worlds: across the six new models, target Spearman is 0.495–0.571, shared fraction is 0.974–0.985, and incremental specific rank-R2R^2 remains at or below 0.0085.

In additional worlds with edge-rewiring probabilities of 0.05 and 0.10 per landscape-tree edge, exact-target shared fractions fall to 0.301 and 0.158. Across two seeds and three scales at each level, K=32K=32 exceeds K=4K=4 in shared alignment, suppression reduces shared alignment, and PLM shared fraction exceeds the corresponding exact-target fraction in all 12 configurations. Numerical rows are reported in Appendix A.3.

Weinstein et al. (2022)

We have added Weinstein et al., Non-identifiability and the Blessings of Misspecification in Models of Molecular Fitness (NeurIPS 2022) and contextualized it in the revised Related Work section. Their result clarifies why observational sequence-density estimation need not identify fitness when phylogeny distorts the observed distribution. Our controlled analysis complements this by asking which component of exact target mutation effects a sequence-density model recovers under heterogeneous contexts.

AuthorsOpenReview ↗

Response to Reviewer q63X (continued)

Summary

Each question raised by the reviewer is answered in the revision. The optimization mismatch and the suppression-centering discrepancy are corrected. All intervention conditions are retrained under one rule per scale. The supplement documents the matched settings. The exact-target and noisy-target references show that the shared-dominant decomposition exceeds the target's own composition. The representation claim is restricted to the tested mechanisms and re-evaluated on the Figure 1 checkpoints. The equal-weight T/A/B comparison led us to withdraw the historical-weight claim. Three results regarding secondary claims in the original submission are narrower and are now stated as such in the revised version. Namely, the breadth effect is mainly first-order, the excess over the exact-target reference is not consistent for the xxxl model at the base budget, and the natural analysis is observational. The principal conclusion of our paper does not depend on these and now further holds across three independently generated worlds, rewired topologies, and a 4×4\times increase in training data.

Reviewer_z7wBOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
This paper asks whether zero-shot mutation scores from protein language models follow constraints shared across training contexts, or the fitness landscape of the specific target. In a synthetic Potts setup, landscapes are arranged in a tree and mutation effects are split into the mean over the training landscapes and a target-specific residual. The authors train small masked transformers, then vary how many landscapes are seen at a fixed sequence budget and shrink the shared part of the Potts parameters. On ProteinGym they fit a similar split to identity-matched homolog clusters and compare it with ESM2 scores and with experimental DMS. They also test whether a lineage that can be read out from hidden states is the lineage the mutation score follows. The reported result is that scores track the shared component, that removing shared structure removes the signal, and that decoded context does not determine the predicted mutation profile. The synthetic evidence is one generated world, with intervals over 16 held-out landscapes. The main natural cohort is 5 of 173 targets, with a 9-target sensitivity set, split-half checks, and a small historical UniRef50 comparison.
Critical strengths

The fixed-budget sweep is a real result. With the number of training sequences held at 5,120, increasing the number of landscapes from 4 to 32 raises the model's correlation with the shared component (0.054 to 0.159 for the smallest model), while the oracle correlation between that component and the target barely moves (about 0.88 to 0.90). This is the main reason I do not score the paper a clear reject.

The authors also separate sequence remoteness from a change of fitness function, and they report the weak spots instead of hiding them: one generated world, 5 of 173 natural targets, and split-half reliability of about 0.89 for the shared term versus 0.33 for the specific term. That makes the paper easier to judge. It is not enough to accept it.

Critical weaknesses

The main result is mostly built into the simulation. A held-out landscape already correlates about 0.91 with the nearest training landscape and about 0.90 with the training average, and the model itself only reaches about 0.13–0.16 against that average. "80–85% shared" is a split of a rank-R^2 around 0.03. The suppression run then drops the target–shared oracle correlation from about 0.90 to about 0.22, while the specific residual stays near zero, where it started. There is no single-landscape run showing that this residual can be learned at all, and no field-only baseline. I cannot tell "the model prefers shared constraints" from "the model only ever fit the easy shared part."

The other settings do not rescue the claim. On the sequences that are actually scored, landscape identity is recovered at about chance (one third); the high decoding accuracy is on the support sequences. The ProteinGym split uses 5 of 173 targets, and its "shared" term is the average of two other clusters in that alignment, not ESM2's training mixture. The specific term's split-half correlation is about 0.33. These are the reasons I reject rather than ask for a rewrite.

Questions for authors
Clarifications, not new experiments. The abstract says lineage is encoded but not used. In Section 4 and Appendix A.2, is the high decoding accuracy only on the classifier support sequences, and is assignment of the remote queries that are actually scored near one third? I want to be sure I have not missed a result where those scored queries are decoded correctly.

The next two answers could change my score. If they are not in the paper, a rebuttal number is enough. A promise of future work would not change the recommendation.

1. If you subtract a site-independent or field-only score, how much of the correlation between the model and the shared component is left?
2. If the same model is trained on one landscape at the same sequence budget, can it fit the target-specific residual r(x)?
8 comments
AuthorsOpenReview ↗

Response to Reviewer z7wB

We thank the reviewer for the detailed review and for identifying the two controls that most directly distinguish preferential recovery of shared constraints from an "easy shared part first" explanation. We ran both requested analyses and added reference decompositions that make this distinction explicit. (Please note that terms are defined in the summary of revisions above.)

Clarification: is the high decoding accuracy only on the classifier support sequences, and is assignment of the remote queries that are actually scored near one third?

Yes, to both. The high decoding accuracy applies to the support sequences only, and assignment of the scored remote queries is near one third. Specifically, the high decoding accuracy is support-sequence separability, not reliable retrieval of the scored remote queries. We reran the representation analysis on the same optimization-matched K=32K=32 checkpoints used for Figure 1 across three worlds, two training seeds, and three scales (18 models). Local support accuracy is 0.968/0.979/0.982 for xs/l/xxxl, whereas local remote-query accuracy is only 0.345/0.361/0.351 (chance = 1/3); in the strict identity-matched subsets it is 0.338/0.347/0.347. Whole-sequence same-wrong margins are small and positive (means 0.0064/0.0100/0.0063; seed-averaged world ranges 0.0030–0.0111, 0.0075–0.0127, and 0.0030–0.0086), which is compatible with weak distributed context use. Masked-site same-wrong margins are +0.0013+0.0013/+0.0002+0.0002/−0.0001-0.0001 for xs/l/xxxl, with the seed-averaged world ranges spanning zero. Training contexts are thus separable in representation space, but local retrieval does not reliably assign a sequence-remote query to the landscape that generated it.

Q1: How much PLM–shared correlation remains after a site-independent or field-only control?

We added two complementary controls. After controlling for the exact Potts field-only contribution, PLM–shared partial rank correlations remain 0.164/0.137/0.124 for xs/l/xxxl.

A stronger empirical per-site profile, estimated directly from the 5,120 training sequences with pseudocount 1, accounts for 78%, 83%, and 83% of the raw PLM–shared correlation at this base budget. The residual partial correlations are 0.040 [0.019,0.059][0.019, 0.059], 0.026 [0.014,0.039][0.014, 0.039], and 0.023 [0.006,0.040][0.006, 0.040], and incremental shared rank-R2R^2 beyond the profile is 0.0062/0.0055/0.0058. We therefore agree that, at 5,120 sequences, the recovered shared signal is predominantly first-order. The same conclusion holds in the two additional fixed-topology worlds: after fitting each world's own 5,120-sequence profile, seed-averaged partial PLM–shared correlations range from −0.007-0.007 to 0.0360.036 and incremental shared rank-R2R^2 beyond the profile from 0.0053 to 0.0082.

This changes at larger budgets. At 20,480 training sequences, complete-target Spearman reaches 0.673/0.668/0.636 and the same profile control leaves PLM–shared partial correlations of 0.703/0.703/0.669. Incremental shared rank-R2R^2 beyond the profile is then 0.421/0.442/0.400. The fitted profile itself correlates only 0.364 with the shared oracle. Models trained on more data thus recover substantial shared structure that an independent-site profile cannot explain. We report the full budget series in Appendix A.3. We also repeated the 20,480-sequence K=32K=32 training in two additional independently generated worlds with the same model seed. Across those six added models, complete-target Spearman is 0.495–0.571, shared fraction is 0.974–0.985, and incremental specific rank-R2R^2 is 0.0051–0.0085. Together with the reference world, all nine 4×4\times models remain shared-dominant as overall prediction improves. The 4×4\times empirical-profile decomposition is evaluated in the reference world only, so we do not treat its numerical magnitude as a replicated world-level effect.

The profile analysis also qualifies the fixed-budget KK-sweep. At 5,120 sequences, the K=32K=32 minus K=4K=4 partial-correlation contrast after profile control is positive for xs (+0.0369+0.0369 [0.0046,0.0697][0.0046, 0.0697]) and xxxl (+0.0211+0.0211 [0.0045,0.0412][0.0045, 0.0412]) but inconclusive for l, while incremental shared rank-R2R^2 beyond the profile does not increase with KK. We therefore ran the same K=4K=4 versus K=32K=32 comparison at 20,480 sequences. The profile-controlled partial-correlation advantage is +0.026+0.026 [0.005,0.047][0.005, 0.047] for xs, +0.008+0.008 [−0.009,0.027][-0.009, 0.027] for l, and −0.0002-0.0002 [−0.0226,0.0186][-0.0226, 0.0186] for xxxl; incremental shared rank-R2R^2 beyond the profile does not differ reliably at any scale. Breadth thus raises raw shared alignment, but evidence that breadth itself increases higher-order incremental shared recovery is limited. The 4×4\times result shows that more data allows both conditions to recover substantial shared structure beyond first-order profiles.

AuthorsOpenReview ↗

Response to Reviewer z7wB (continued)

Q2: Can the same model learn the target-specific residual when trained on one landscape at the same sequence budget?

Yes. We trained the same architectures on each of eight individual landscapes with the same total 5,120-sequence budget. To preserve broad genealogical sampling without mixing fitness functions, each specialist uses 32 independently generated sequence-tree replicates under its single governing landscape. Specialist and pooled K=32K=32 models are evaluated on the same 128 near and 128 remote queries.

The raw specialist-minus-pooled PLM–specific Spearman differences are +0.218+0.218/+0.217+0.217/+0.231+0.231 on near queries and +0.322+0.322/+0.314+0.314/+0.315+0.315 on remote queries for xs/l/xxxl. Because the shared and residual predictors are correlated, we also computed the paper's primary incremental rank-R2R^2 endpoint. Specialist-minus-pooled incremental specific rank-R2R^2 is +0.065+0.065 [0.024,0.109][0.024, 0.109], +0.039+0.039 [0.016,0.064][0.016, 0.064], and +0.049+0.049 [0.017,0.084][0.017, 0.084] on near queries, and +0.058+0.058 [0.035,0.080][0.035, 0.080], +0.044+0.044 [0.031,0.057][0.031, 0.057], and +0.045+0.045 [0.031,0.060][0.031, 0.060] on remote queries. Every interval is above zero.

The specialist shared fractions are 0.698/0.732/0.710 on near queries and 0.738/0.763/0.766 on remote queries, compared with exact-target fractions of 0.756 and 0.763 on those same query sets. The pooled model is more shared-dominant (0.915/0.840/0.836 near and 0.885/0.856/0.830 remote). The same architectures can thus recover incremental target-specific structure when trained on one landscape; the weak residual under heterogeneous training is not an intrinsic inability to fit it. These results are in Appendix A.3.

Suppression: does making the target mostly specific expose a specific predictor?

No. The training-centered intervention reduces the training common-variance fraction from 0.700 to 0.044 in the reference world. We distinguish two oracle diagnostics. Evaluating the original and transformed parameter banks on the same baseline query backgrounds gives target–shared similarity 0.898→0.4810.898 \to 0.481, which isolates the parameter manipulation. The suppression experiment regenerates queries under the transformed landscapes; on that evaluation cohort, target–shared similarity is 0.898→0.3240.898 \to 0.324 and the exact-target shared fraction falls from 0.775 to 0.028.

Although the target is then almost entirely landscape-specific, PLM–target Spearman falls from 0.156/0.130/0.119 to 0.039/0.029/0.025. PLM–shared Spearman falls from 0.182/0.155/0.137 to 0.049/0.037/0.022, while PLM–specific Spearman remains small at 0.032/0.024/0.022. Across three worlds and two training seeds, the specific response to suppression averages +0.0003+0.0003/+0.0010+0.0010/−0.0034-0.0034. Combined with the specialist result above, this shows that the residual is learnable by these architectures, but pooled heterogeneous training does not recover it when shared structure is removed.

Is shared dominance built into the decomposition or produced by low prediction accuracy?

No to both. Applying the identical decomposition to the exact target mutation vector gives a shared fraction of 0.775 (95% landscape-bootstrap CI [0.745,0.806][0.745, 0.806]). The matched K=32K=32 PLM fractions are 0.886/0.861/0.837 for xs/l/xxxl; the paired PLM-minus-target excess is +0.111+0.111 [0.072,0.141][0.072, 0.141], +0.086+0.086 [0.048,0.119][0.048, 0.119], and +0.062+0.062 [0.009,0.108][0.009, 0.108].

We added an accuracy-matched noisy-target reference. Gaussian noise was added to the exact target until its target Spearman matched each PLM (1,000 draws). The resulting noisy-target shared fractions are 0.738 [0.712,0.762][0.712, 0.762], 0.726 [0.696,0.757][0.696, 0.757], and 0.720 [0.687,0.751][0.687, 0.751], below the corresponding PLM fractions of 0.886/0.861/0.837. We repeated this control for all 12 matched K=32K=32 checkpoints in the two additional fixed-topology worlds. PLM shared fraction exceeds the accuracy-matched noisy reference in every added checkpoint, by 0.056–0.203. Thus, low target correlation alone does not produce the observed shared fraction.

The excess is largest where the target is least shared. With per-edge rewiring probabilities 0.05/0.10 at each landscape-tree transition, exact-target shared fraction is 0.301/0.158, while PLM shared fractions are 0.56--0.71/0.51--0.61 across all 12 configurations (Appendix A.3). A decomposition driven only by target composition would not produce this separation.

Using the true target landscape only as an oracle, we formed kk-nearest mixtures of training landscapes by tree or Potts-parameter distance. Across k∈{1,2,4,8,16}k \in \{1, 2, 4, 8, 16\}, two distance definitions, three scales, three worlds, and two training seeds, all 180 nonuniform mixtures are closer to the exact target but less correlated with PLM predictions than the uniform 32-landscape mean; the uniform mean contributes more incremental PLM rank-R2R^2 in 178/180 joint comparisons (Appendix A.3). The uniform shared result is therefore not a poor proxy for a more target-appropriate local interpolation.

AuthorsOpenReview ↗

Response to Reviewer z7wB (continued)

Natural cohort size and reliability

We agree that five strict targets do not support a population-level estimate, and we do not make one. We also agree that the natural S=(A+B)/2S=(A+B)/2 estimator is not ESM2's training mixture. It is an identity-controlled within-family estimator of constraints recurring across homolog clusters. Selection is outcome-blind. Shared exceeds specific in all 30 target-by-scale comparisons in the strict cohort and all 54 in the nine-target sensitivity cohort. The split-half reliability of the specific residual (0.334) is lower than that of shared (0.891), but after errors-in-variables correction the ESM2 shared increment remains 0.097–0.159 versus 0.037–0.061 for specific. The natural analysis is observational and is used only to establish a concordant within-family ordering.

We also tested the reviewer's concern that natural SS averages only two comparison clusters. Strict matching permits m=3m=3 for only two primary-cohort targets, so we do not claim a population-level breadth curve. On those two targets, however, adding a third comparison cluster increases incremental shared rank-R2R^2 and decreases incremental specific rank-R2R^2 in all 12 target-by-ESM2-scale comparisons (expanded Appendix C.6).

Summary

Summary

The review isolated two questions whose answers the reviewer stated could change the assessment, indicating that a rebuttal number would suffice. Our revision provides both.

  • Q1 (site-independent control). At the base budget, we agree with the reviewer. An empirical per-site profile accounts for approximately 80% of the PLM–shared alignment at 5,120 sequences. This first-order regime does not persist. At 20,480 sequences, the models recover shared structure that the profile does not explain (partial correlations of 0.67–0.70; incremental shared rank-R2R^2 of 0.40–0.44 beyond the profile).
  • Q2 (single-landscape control). The same architectures recover the target-specific residual when trained on a single landscape at the same sequence budget. All intervals for incremental specific rank-R2R^2 exclude zero, and the specialist shared fraction matches that of the exact target. The weak residual under pooled training therefore reflects heterogeneous training rather than limited capacity.

We tested the reviewer concern that shared dominance is built into the simulation against three references, and the PLM shared fraction exceeds each: the exact target, an accuracy-matched noisy target, and topology-rewired worlds in which the exact target is only 16–30% shared while the models remain 51–71% shared.

The controls requested by the reviewer strengthen the central conclusion of our paper. Under heterogeneous training, the models preferentially recover shared constraints over target-specific structure that the same architectures can learn. This result replicates in three independently generated worlds, under rewired interaction topologies, and at 4×4\times data, where target Spearman reaches 0.50–0.67 and incremental specific rank-R2R^2 remains below 0.01 for all nine models.

The revision also narrows three claims in line with the review: the breadth effect is mainly first-order at the base budget, the representation claim is restricted to the tested mechanisms, and the natural analysis is presented as observational. The central conclusion does not depend on these secondary claims. It depends on the single-landscape control, the suppression intervention, and the data-scaling result. We thank the reviewer for the two controls.

Reviewer_Dy89OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether zero-shot mutation scores from protein language models track constraints shared across evolutionary contexts or constraints specific to the target. A synthetic hierarchy of Potts landscapes supplies exact oracles, letting mutation effects be split into a training-ensemble mean (shared) and a target residual (specific); drop-one incremental rank-R² then attributes model scores to each. Two interventions follow: sweeping the number of training landscapes at a fixed 5,120-sequence budget, and suppressing shared Potts structure while preserving parameter RMS. A natural analogue fits Potts models to identity-matched homolog clusters and scores ESM2 at six scales. Probing tests whether decodable cluster identity governs the predicted mutation profile. The reported result is that scores follow the shared component, with 80–85% of summed attributable contributions.

Critical strengths

The question is well posed, and the synthetic setup is the right tool for it. Exact oracles, plus the separation of a landscape tree from a sequence tree, let the authors tell a real change of fitness function apart from ordinary sequence remoteness, which you simply cannot do with observational corpora. The two interventions also go beyond correlation. Holding the budget at 5,120 sequences while raising the number of landscapes from 4 to 32 lifts shared alignment from 0.054 to 0.159 at the smallest scale, while the shared–target oracle correlation barely moves. And suppressing shared structure at fixed parameter RMS wipes out the signal instead of exposing a specific predictor. I also appreciate that the paper reports its own weak spots: single-world intervals, 5 of 173 natural targets, and split-half reliability of 0.891 for shared against 0.334 for specific.

Critical weaknesses
  1. The natural experiment does not measure the same thing as the synthetic one. In the synthetic setting, the shared component is the mean over the 32 landscapes the model was actually trained on, so "shared" really does mean common to the contexts seen during training. In the natural setting it becomes S = (A+B)/2 (defined at line 213), just the average of 2 sibling homolog clusters taken from the target's own MSA (Section 3.4). Two things follow. First, averaging 2 clusters is a coarse way to estimate what is common across a family, when the synthetic version averaged 32. Second, and more importantly, S cannot stand in for ESM2's training distribution. That distribution is UniRef50, which contains T, A and B along with everything else and cannot be enumerated or partitioned the way the 32 synthetic landscapes can. So while the synthetic setting measures what is common across the contexts the model was trained on, the natural setting measures something narrower, namely what is common within one family. I would ask the authors to state this scope clearly, and to show how S behaves as more comparison clusters are added.

  2. The causal evidence does not come from protein language models. Both interventions need control over the training corpus, so they can only be run on the masked Transformers the authors train themselves: 46k, 667k and 6.33M parameters, trained on 5,120 synthetic Potts sequences. ESM2 only shows up in correlational analyses and is never intervened on. Besides, The gap of scale raises concern about whether the conclusion obtained on the small model can be tranferred to larger model, considering the size of ESM ranges from 8M to 15B. Please either scope the causal conclusions to the controlled system, or add an intervention that is feasible on a real PLM.

Questions for authors

Please see the above weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Dy89

We thank the reviewer for their positive assessment and for identifying two important scope distinctions. We agree that the natural S=(A+B)/2S=(A+B)/2 estimator is not the same object as the synthetic training-ensemble mean, and that causal conclusions should be restricted to systems on which we intervene. (Please note that terms are defined in the summary of revisions above.)

Natural SS is a within-family estimator, not ESM2's pretraining distribution

In the synthetic system, the shared component is exactly the mean over the 32 landscapes used to train the model. In ProteinGym, S=(A+B)/2S=(A+B)/2 instead estimates constraints recurring across identity-matched operational homolog clusters within one family. It cannot be interpreted as an enumeration or partition of ESM2's full UniRef50 pretraining distribution. We have revised Secs. 3.4 and 4.3 and Appendix C.6 to call this explicitly a within-family estimator of reusable constraint and not ESM2's training mixture.

This distinction is also consistent with our historical-UniRef analysis. Adding the target cluster to an equal comparison mixture helps, but weighting T/A/B by historical UniRef proportions does not improve over equal T/A/B. We therefore no longer use the historical result to imply reconstruction of ESM2's training mixture.

Sensitivity to adding more comparison clusters

We implemented the suggested sensitivity by defining SmS_m as the mean of mm nested, outcome-blind comparison-cluster Potts vectors while retaining the original identity-matching criteria and one jointly matched homolog support within each target. Feasibility itself is limiting: all nine targets support m=2m=2, only two primary-cohort targets support m=3m=3, and one supports m=4m=4 or 55. We therefore do not interpret the larger-mm rows as a population-level scaling curve.

On the two primary-cohort targets where m=3m=3 is feasible, however, the direction is consistent across every ESM2 scale. From S2S_2 to S3S_3, mean incremental shared rank-R2R^2 rises 0.169→0.2340.169 \to 0.234, incremental specific rank-R2R^2 falls 0.0234→0.00990.0234 \to 0.0099, PLM–shared Spearman rises 0.435→0.4980.435 \to 0.498, and PLM–specific Spearman falls 0.204→0.1380.204 \to 0.138. Incremental shared increases and incremental specific decreases in all 12 target-by-scale comparisons. The experimental DMS endpoint shows the same change on both targets (its shared fraction increases from 0.8180.818 to 0.9820.982 for CCR5 and from 0.9820.982 to 0.9970.997 for VG08.). Thus we find no evidence that shared dominance is created specifically by averaging only two comparison clusters while explicitly retaining the small-cohort limitation (Appendix C.6).

Scope of the causal conclusion

We take the reviewer's first suggested remedy and restrict the central causal claim to the controlled synthetic system. The KK-sweep manipulates training breadth and establishes an effect on raw shared alignment, but the empirical-profile controls show that its base-budget effect is mainly first-order. Therefore, we do not claim that breadth alone increases higher-order recovery. Shared suppression, single-landscape specialization, data scaling, independent-world replication, and topology rewiring likewise manipulate the training distribution of the masked transformers models that we train ourselves. The ESM2 checkpoints from 8M to 15B are evaluated observationally. We do not intervene on their original UniRef50 pretraining distribution. The revised Abstract, Results, Limitations, and Conclusion sections now make that distinction explicit, and the ESM2 result is described as qualitatively concordant observational evidence rather than a causal intervention on ESM2.

There is nevertheless a separate natural-sequence training intervention already in Appendix C.11. For 23 ProteinGym targets spanning 19 CATH homologous superfamilies, we train xs and l models on 128-sequence cohorts drawn either narrowly from one 35%-identity Gene3D cluster or broadly from at least eight clusters, with exactly matched target-identity distributions. Broad training improves DMS Spearman by +0.076+0.076 (95% CI [0.047,0.106][0.047, 0.106]) for the xs model and +0.097+0.097 ([0.063,0.135][0.063, 0.135]) for the l model. This experiment measures mutation-prediction accuracy rather than the shared/specific decomposition and uses small trained models. So we treat it as causal support for the direction of a breadth effect on natural sequence data rather than evidence that the same mechanism causally governs pretrained ESM2.

Summary

Both points raised by the reviewer are addressed in the revision. The natural estimator is now defined and described as a within-family quantity, with the requested sensitivity to additional comparison clusters, and the causal claims are restricted to the controlled system. The central result is unchanged and now holds across three independently generated worlds, rewired topologies, and a 4×4\times increase in training data.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions in response to reviewer feedback

We thank the reviewers for their constructive comments. We here summarize our revisions for all, in addition to responding below to each reviewer separately.

Added. We retrained the KK-sweep and suppression conditions under one optimization rule per scale, with suppression centered on the training landscapes. We added the following controls:

  • exact-target and accuracy-matched noisy-target references
  • field-only and empirical-profile controls
  • single-landscape specialists at the same sequence budget
  • 4×4\times data scaling, with a matched K=4K=4 control and replication in two additional worlds
  • replication of the central interventions in three worlds with two seeds
  • topology rewiring
  • representation analyses on all 18 matched checkpoints
  • an equal-weight T/A/B historical baseline
  • a third comparison cluster for the natural analysis, where feasible

Narrowed. Causal claims are confined to the controlled system. Breadth increases raw shared alignment, but at the base budget this effect is largely first-order and does not consistently increase incremental shared rank-R2R^2 beyond the empirical profile. The representation claim is restricted to the tested context-selection mechanisms: whole-sequence assignments show a small context-aligned effect, whereas masked-site retrieval is unreliable and correct local retrieval does not increase target-specific alignment. The historical-weight claim has been withdrawn.

Retained. Under heterogeneous training, the recovered signal is more shared than the target itself. The model's shared fraction exceeds the exact-target shared fraction in 15 of 18 configurations in fixed-topology worlds (three worlds, two seeds, three scales) and in all 12 configurations in rewired worlds. Single-landscape specialists recover additional target-specific signal. At 4×4\times data, the same shared-dominant decomposition appears in all nine models (three fixed-topology worlds, three scales): shared fractions are 0.974–0.992, against 0.702–0.775 for the exact target, and incremental specific rank-R2R^2 is at most 0.0085.

Terminology. We use “experimental configuration” for one world–seed–scale combination. The synthetic scales xs/l/xxxl contain 46k/667k/6.33M parameters. We use “fixed-topology worlds” for independently generated worlds with the original interaction topology. In response to reviewers, we also generate worlds with a nonzero edge-rewiring probability: at each landscape-tree transition, each existing interaction edge is independently rewired with probability 0.05 or 0.10. Rewiring accumulates along the tree, so the mean fraction of interaction pairs shared by two terminal landscapes falls to 0.320 and 0.179, respectively.

The supplement documents the matched experimental settings and all added controls.

Summary. The central conclusion of our original submission is unchanged, and the added controls actually strengthen it. Specifically, under heterogeneous training, the models recover constraints shared across contexts in preference to target-specific structure, although the same architectures learn that structure when trained on one landscape. This holds across three worlds, rewired topologies, and a 4×4\times increase in data. The narrowed claims concern the mechanism of the breadth effect, the representation analysis, and the scope of the ESM2 evidence. The central conclusion does not depend on any of them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.