acceptodds
Under review as a conference paper at ICLR 2027

PROPEL: Global Evolutionary Latents for Autoregressive Protein Language Models

Abstract

Autoregressive protein language models capture rich sequence regularities, but represent protein family information implicitly through shared model parameters and token-level hidden states. Other family-specific models about one family are spread across weights shared with every other family, so pointing it at a new target means changing the model. We introduce PROPEL (Protein Retargeting by Optimized Posterior over Evolutionary Latents), an autoregressive protein language model with latent space that complements local autoregressive modeling with a compact global representation of evolutionary context. A target-specific latent state is shared across the sequence and injected into every decoder layer through cross-attention, separating persistent family-level information from token-level sequence computation.The latent is learned without an encoder and can be further optimized at inference time while the backbone remains frozen. This representation provides a flexible interface for incorporating both unlabelled homologs and limited fitness supervision, while preserving a single autoregressive model for both variant scoring and sequence generation. Across protein fitness prediction tasks, PROPEL consistently improves its autoregressive backbone and exhibits strong data and parameter efficiency: a 60M-parameter model can match substantially larger autoregressive protein language models under zero-shot fitness prediction and comparable alignment settings. These results show that introducing an explicit global representation can substantially enhance autoregressive protein models without sacrificing their simplicity, scalability, or generative nature.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
protein language models, protein design, latent-variable models, variational inference, test-time adaptation, protein fitness prediction, parameter-efficient adaptation, controllable generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_i2kvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PROPEL, a 60.1M-parameter autoregressive protein language model with a target-specific latent state. Homologs and fitness measurements can update the same latent while keeping the decoder frozen, and the resulting state is used for both variant scoring and sequence generation.

Critical strengths
  1. The writable latent provides a unified interface for incorporating both homolog information and fitness measurements without modifying the target-specific backbone weights.

  2. The paper provides extensive analyses of latent inference, supervised adaptation, storage/compute trade-offs, and conditional generation.

Critical weaknesses
  1. The necessity of the dedicated latent architecture is not established. PROPEL requires a specially pretrained 60M model with substantial latent-specific machinery, yet Table 5 reports only 0.360 in the label-free setting, essentially identical to the Site-Independent homolog profile (0.359), below ESM-2-650M using only the query sequence (0.414), and substantially below existing homolog-conditioned methods (0.432-0.479). Comparisons with strong pretrained PLMs under standard adaptation are necessary.

  2. The matched-budget adaptation comparison does not isolate the benefit of the latent representation. The latent is accessible across all 12 decoder layers, whereas the 12,288-parameter LoRA baseline is applied only to the final attention output projection. Besides, distributed LoRA and matched per-layer prefix controls are not evaluated.

Questions for authors

How does PROPEL compare with existing pretrained protein language models using standard LoRA or prefix adaptation?

3 comments

We thank the reviewer for the thoughtful comments and helpful suggestions. We appreciate the opportunity to clarify our approach and provide detailed responses to the points raised below.

The requested stronger-backbone adaptation comparison is complete. PROPEL latent achieves 0.4538 ± 0.0212 versus 0.3679 ± 0.0179 for ProGen3 LoRA at 100 updates. The paired difference is +0.0859 [0.0389, 0.1374]; the equal-family sensitivity is +0.0768 [0.0433, 0.1102].

This is the additional 128-label protocol described in the common response. The methods share label subsets, validation roles, and fixed test subsets of up to 4,096 examples per assay. PROPEL uses five context-conditioned states and 12,288 adapted values; ProGen3 uses query-only input, bidirectional scoring, and 139,264 adapted values. Table 4's published ProGen3 rows instead report full-weight IRPO under the original full-label/full-test protocol; they are separate comparisons.

All 24 external assay/seed cells use the corrected deterministic runs. Together with distributed LoRA, activation refinement, and per-layer prefixes, these controls address the missing-baseline objection while separating the benefit of the state interface from a claim that latent coordinates are uniquely necessary.

Thanks for the extra experiments. However, my main concerns about whether the proposed latent architecture is truly necessary, and whether the adaptation baselines were compared fairly, haven't been resolved.

Thank you for the follow-up. We appreciate the distinction between demonstrating useful performance and establishing why a particular architecture is needed. We would like to clarify our technical contribution and the scope of the additional comparisons.

The contribution of the latent architecture. Our aim is to demonstrate the usefulness of an explicit, reusable latent state and competitive performance in the evaluated adaptation setting. The comparison with strong pretrained autoregressive models places these capabilities in context; we do not claim general superiority over those models.

PROPEL provides a variational learning-and-inference mechanism through which unlabeled context constructs a compact, reusable state and limited supervision refines that same state. A frozen decoder consumes the resulting representation for prediction and generation. We demonstrate this capability with a 60.1M-parameter model pretrained on 13.5B non-padding tokens, adapting only 12,288 state values with supervision.

Two results directly support the utility of this mechanism. Increasing inference steps from 16 to 128 improves grouped correlation from 0.2089 to 0.3595, with gains on 189/217 assays, while keeping the checkpoint and context fixed. On the original full-label protocol, combining context and supervision achieves 0.5182, compared with 0.3545 for context alone and 0.3184 for supervision alone.

These controls illustrate the flexibility of the latent-state interface. Activation and prefix adaptation retain the variationally inferred conditioning state and the pretrained decoder, while introducing supervision through different correction mechanisms. Their competitive performance supports using the same inferred representation with latent updates, activation residuals, or per-layer prefixes while keeping backbone weights frozen. The architecture therefore accommodates multiple ways to incorporate supervision into a state-conditioned model.

Fairness of the adaptation comparisons. The original comparison left an important depth-coverage asymmetry: latent refinement acts across all layers, whereas the original LoRA control acts only at the final layer. We addressed this specific concern by adding rank-1 LoRA across all 12 layers.

The native arms share the pretrained backbone, inferred-state conditioning, 12,288 adapted values, 128 available labels including validation, 100 updates, and fixed test subsets. Each method independently selects its learning rate from eight candidates using validation data. The reported results average three seeds varying label subsets and optimizer order.

Latent minus comparatorPaired difference, 95% interval
All-layer LoRA+0.0569 [0.0194, 0.0997]
Final-layer LoRA+0.0323 [0.0190, 0.0478]
Activation refinement−0.0050 [−0.0211, 0.0134]

Seeds are averaged within assay before paired resampling. Both LoRA contrasts also remain positive under equal-family weighting. Thus, the advantage over the tested LoRA configurations persists with updates distributed across layers and independent learning-rate selection. The external ProGen3 comparison additionally differs in context inputs and adapted capacity, so we treat it as supporting context rather than an architectural ablation.

Our central claim is therefore a reusable variational inference-and-refinement pipeline with demonstrated information-combination and inference-time benefits, together with effective compact adaptation under the specified limited-label protocol. We sincerely hope these clarifications and additional results address your concerns. Thank you again for your time and constructive feedback.

Reviewer_GDtGOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

Protein Models have been a cornerstone application of sequence models. The paper proposes to augment the model class with a latent space that provides a global representation to the local autoregressive modeling. The latent is learned and can be further optimized at inference time which is a very neat idea. This idea is well studied across several tasks. It does not necessarily outperform other approaches but the idea is actually very elegant and the results are very nicely reported. The paper is very polished and so far the strongest in my batch of reviews.

Critical strengths

This is a very neat technical idea explored in detail for an application! The paper is very polished and actually takes some time to process because it is actually dense in information and not just AI slop. It is really great to see "We release the pretrained checkpoint, the family-packed block construction, the posterior inference and write code, and the per-assay states behind every arm in this paper, so that the latent space itself—not only the numbers read off it—is the artifact."I hope the authors will do that but that is actually very nice to see and read!

Critical weaknesses

The references are a mess and should be cleaned up. The abstract reads a bit better than what the experiments then actually show. Nevertheless I think all results are fairly reported and the extended limitations in the appendix is actually very good! This is very nice from the authors!

Questions for authors

I think the paper is nice and actually a valid technical and elegant idea explored in detail. The results are not spectacular but on the other hand they are extensive and fairly reported. I am not directly sure what would significantly improve or change the paper.

1 comment

We appreciate the reviewer for recognizing the reusable interface and the breadth of the study. We lead the abstract and discussion with parameter and data efficient adaptation from a compact model pretrained on 13.5B tokens, supported by complementary context and supervision, useful inference-time refinement, and the completed 128-label, 100-update comparisons.

The corrected bibliography is uploaded, including repaired author lists and consistent proceedings metadata.

Reviewer_4j6YOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PROPEL, an autoregressive protein language model with approximately 60M parameters and global evolutionary latent variables. The model is pretrained on blocks of homologous sequences from the same family. It learns a 24 x 512 latent state through encoder-free iterative variational inference and injects this state into each decoder layer through cross-attention.
The experiments cover 217 ProteinGym substitution assays and eight tasks involving adaptation on near-wild-type variants and evaluation on more distant variants. The results show that homologous information, inference-time optimization, and experimental supervision can each improve prediction. Under the tested parameter and update budgets, latent adaptation outperforms the specific LoRA control and can also alter the generation distribution.

Critical strengths
  1. A unified and compact interface for target adaptation.
    Unlabelled homologous information and experimental supervision can update the same state, while scoring and generation share the same decoder. This makes target-specific information an object that can be stored, reused, and further optimized. Compared with training only a property-prediction head, this interface directly influences the model’s sequence probability distribution and therefore supports a broader range of uses.
  2. The experiments demonstrate the benefit of additional latent optimization within the current pipeline.
    The authors hold the checkpoint, homologous context, and scoring procedure fixed while increasing only the number of posterior optimization steps. Across 217 assays, increasing the step count from 16 to 128 raises grouped Spearman correlation from approximately 0.209 to 0.360, with improvements on 189 assays. This design provides relatively direct evidence that additional inference-time compute can improve the current model’s family representation without updating backbone parameters.
  3. The experiments demonstrate complementarity between evolutionary information and fitness supervision.
    Across eight supervised tasks, the authors compare homologous information alone, fitness labels alone, and their combination. Combining both sources increases the mean correlation on distant variants from approximately 0.355 to 0.518, with improvements on all eight assays.
Critical weaknesses
  1. The contributions of latent-variable modeling and a general conditioning interface are not sufficiently separated.
    In Table 3, direct activation-space optimization achieves 0.5243, with no clear difference from the 0.5182 achieved by latent adaptation. This suggests that the supervised adaptation gains may primarily arise from the compact cross-layer conditioning interface rather than the Gaussian prior, KL regularization, or latent coordinates themselves. This result does not undermine the role of variational inference in constructing family states without labels. However, the paper should distinguish which benefits depend on latent learning and inference and which can be obtained through ordinary optimizable prefixes or conditioning states.
  2. The LoRA control matches parameter counts but not intervention locations.
    PROPEL outperforms the parameter-matched LoRA configuration used in the paper, but this does not establish that latent adaptation generally outperforms LoRA. A LoRA control covering all 12 layers would help distinguish the effects of the adaptation mechanism from those of its placement.
  3. The zero-shot results do not establish that the model learns evolutionary information beyond simple statistical models.
    In Table 5, PROPEL achieves a grouped Spearman correlation of 0.360, compared with 0.359 for the Site-Independent model constructed from the same source alignments. The reported results do not clearly distinguish the two. Thus, the paper has not demonstrated that its more complex latent-variable model extracts useful information beyond first-order site frequencies on this evaluation.
    The comparison with ProGen also involves different inputs: PROPEL uses homologous sequences, whereas the reported ProGen results use only the query sequence.
  4. The generation experiments support distributional control, but evidence for functional design remains limited.
    Increasing probability mass in high-fitness regions of a measured variant library is not equivalent to generating novel, highly functional sequences. The decoded-sequence evaluations primarily rely on ESM-1v scores, compositional similarity, and the decoder’s own family preference.
    The appendix demonstrates that latent conditioning can change the outputs, but more rigorous validation of output validity and functional gains is still needed.
Questions for authors
  1. What is the independent contribution of latent-variable modeling relative to an ordinary conditioning state?
    Direct activation-space adaptation and latent adaptation perform similarly. Please separately evaluate the roles of variational inference, KL regularization, the prior network, and family-packed pretraining, and clarify which conclusions concern latent-variable modeling and which concern the cross-layer conditioning interface.
  2. Can you provide parameter-efficient adaptation controls with matched intervention locations?
    Please compare rank-1 LoRA across all 12 layers, per-layer prefixes, and the current final-layer rank-12 LoRA under the same parameter budget. Does the main performance ordering persist with longer optimization and independently tuned hyperparameters?
  3. How can you establish that the family state provides information beyond simple site frequencies?
    Please report paired differences and uncertainty relative to the Site-Independent model, stratified by single versus multiple substitutions and homologous-data depth. Could you include a homologous-input control that preserves site frequencies while disrupting inter-site relationships?
  4. Do the generated sequences exhibit verifiable functional improvements?
    Please reevaluate validity, duplication rates, and diversity with outputs restricted to canonical amino acids. Where possible, use experimental measurements or independent functional evaluations to compare adapted, unadapted, and shuffled-label conditions. Which conclusions concern only changes in the generation distribution, and which support functional improvement?
  5. Are the conclusions robust on unseen families and independently pretrained models?
    Please report overlap between the targets and the pretraining corpus, and consider family-held-out evaluation or decontaminated subsets. Can the main trends be reproduced using an independently pretrained checkpoint, distinguishing robustness to adaptation randomness from robustness of the full training pipeline?
2 comments

We thank the reviewer for the careful assessment and constructive questions. We report additional experiments and clarify what they establish.

Q1.

We agree that latent-variable modeling and a conditioning interface should be distinguished. PROPEL infers an approximate posterior from unlabeled context under the decoder's pretraining ELBO, with model weights frozen at deployment. Once inferred, the representation u=Uθ(z)u=U_\theta(z) functions as the decoder’s conditioning state. The distinctive role of latent-variable modeling is therefore how that state and decoder are learned, and how the state is inferred from unlabeled context. The cross-layer interface determines how the resulting information is consumed and refined.

The requested components have distinct roles within this construction:

ComponentRole in PROPEL
Variational inferenceConstructs an approximate posterior over states from unlabeled context.
Gaussian prior and KLSpecify a shared reference distribution and penalize posterior departures from it; the chosen Gaussian parameterization permits a closed-form KL.
Learned transportMaps inferred variables into the conditioning representation used by the decoder.

We attribute adaptation benefits to the learned, reusable conditioning interface and the modeling contribution to variational training and label-free inference. Isolating these effects is nontrivial because the components are coupled within the chosen variational formulation and learned conditioning pathway.

Q2. Matched placement, independent tuning, 100 updates

PROPEL latent achieves 0.4538 ± 0.0212 versus 0.3679 ± 0.0179 for ProGen3 LoRA. The paired difference is +0.0859 [0.0389, 0.1374]; equal-family sensitivity is +0.0768 [0.0433, 0.1102].

Q3. Beyond site frequencies

On 24 metadata-selected assays, including 16 additional targets, the intervention preserves per-site counts and packing invariants while disrupting inter-position relationships. Natural heldout sequences are excluded from inference context.

Endpoint (nats/token)Equal-assay difference [95% interval]Equal-family difference [95% interval]
Heldout natural-sequence NLL gain+0.0701 [0.0239, 0.1210]+0.0508 [0.0095, 0.0941]
Joint-preference contrast+0.1870 [0.1475, 0.2265]+0.1673 [0.1299, 0.2047]

Intact context improves heldout prediction. The joint contrast measures the original-context state's additional preference for natural over frequency-matched permuted heldout sequences, relative to shuffled-context states. These results support conditional information beyond site counts, without identifying a unique mechanism or separating newly inferred relationships from pretrained knowledge.

Q4. Functional improvement

The audit covers 27,648 draws across three conditions and three sampling policies. All 3,072 draws per condition/policy combination are nonempty and alphabet-valid; alphabet validity is enforced by sampling and establishes format validity only. Mean unique fractions within each 128-draw cell are:

PolicyContext-onlyAdaptedShuffled labels
Unrestricted start1.00001.00001.0000
Template-constrained0.99540.96550.9521

On a fixed pool with experimental measurements, we assess whether adaptation shifts probability toward higher-performing outputs:

Adapted minusTop-decile mass difference [95% interval]Expected measured-fitness percentile difference [95% interval]
Context-only+0.0471 [−0.1560, +0.3038]−0.0215 [−0.1244, +0.1029]
Shuffled labels+0.1882 [+0.0412, +0.3663]+0.1627 [+0.0486, +0.2865]

Both differences versus shuffled labels exclude zero, supporting label-dependent functional steering on measured support. Improvement over context-only remains unresolved.

Q5. Corpus overlap and transfer

Targets were selected by exposure evidence and input availability, independently of model performance. Among 217 benchmark entries, the audit found 63 exact matches and detected relationships for 200 in the primary search or 201 across both searches, leaving 16 entries/15 distinct targets without detected hits. Their context-state-minus-zero gain is +0.2559 [+0.2077, +0.2991].

On eight predefined targets, one-context inference improves over zero state by +0.2324 [0.1803, 0.2858], with gains on all eight. In the stricter seven-target subset with no hits in either search, five-context state-minus-zero gain is +0.2712 [0.2259, 0.3100], positive on all seven. This excludes the target with recorded partial overlap.

Q3 tests information beyond matched site counts; Q5 shows that state benefits persist on the evaluated low-detected-overlap targets, including the stricter subset.

Thank you for the author's response. However, I still believe that the evolutionary latent variables proposed in this paper do not yield sufficiently significant results. I will maintain my original rating.

Comments on the paper

2 comments

Thank you for the constructive reviews. Our central claim is parameter- and data-efficient target adaptation from a compact pretrained model. PROPEL has 60.1M model parameters and is pretrained on only 13.5B non-padding tokens .

Target adaptation freezes the model weights and optimizes just 12,288 state values, about 0.020% of the model parameter count. The new controls demonstrate effective adaptation with 128 labeled examples per assay, including validation, outperforming the tested LoRA alternatives under the stated conditions. Context constructs a reusable state, measurements refine it, and one frozen decoder consumes it for prediction and generation.

Combining information improves prediction. On the paper's original eight-assay protocol (Table 4, latent row), context plus labels achieves mean Spearman 0.5182, versus 0.3545 for context alone and 0.3184 for labels alone. The paired gains are +0.1637 [0.0839, 0.2743] and +0.1998 [0.1260, 0.2687], with gains on all eight assays. Three optimizer-order seeds give 0.5142 ± 0.0035 (sample SD); every seed improves over context alone on every assay.

More inference computation improves a fixed model. Holding checkpoint, context, and scoring fixed, increasing state-inference steps from 16 to 128 raises grouped correlation from 0.2089 to 0.3595, improving 189/217 assays. This demonstrates useful inference-time refinement without updating backbone weights.

The additional 128-label, 100-update controls favor PROPEL over the tested LoRA configurations. We use the same eight parent splits, with 128 available labels per assay including validation, versus 532–2,167 labels in the paper's original protocol. Evaluation uses fixed test subsets of up to 4,096 examples per assay, shared across methods and seeds; Table 4 uses complete test partitions. Each method selects its own learning rate from eight candidates using validation data, then refits on all 128 labels for 100 updates.

The table below averages three seeds that vary both labeled subsets and optimizer order. Table 4's 0.5182 is the original single-seed result, with optimizer-only replications reported above. These panels answer different questions under different protocols, so their absolute scores are not directly comparable.

MethodMean SpearmanSD across seed means
PROPEL latent0.45380.0212
PROPEL activation refinement0.45880.0143
PROPEL all-layer LoRA0.39690.0255
PROPEL final-layer LoRA0.42150.0048
ProGen3 LoRA, corrected deterministic runs0.36790.0179

The paired gains over both native LoRA controls and ProGen3 LoRA have positive 95% intervals; the reviewer-specific replies give the contrasts and matching conditions. All intervals are exploratory and conditional on the fixed checkpoint and evaluated tasks; family-group sensitivities combine known related tasks.

Why activation refinement is competitive. Both methods start from the same inferred state zjz_j and use the fixed transformation UθU_\theta and decoder. Their conditioning inputs are

u_j^{A}=U_\theta(z_j)+\Delta u.$$ For a fixed state, the activation correction $$\Delta u_j(\Delta z)=U_\theta(z_j+\Delta z)-U_\theta(z_j)$$ exactly reproduces the latent-corrected decoder input. For small corrections where $U_\theta$ is smooth, $$\Delta u_j(\Delta z)=J_{U_\theta}(z_j)\Delta z+ \mathcal O(\|\Delta z\|^2).$$ This gives a mathematical explanation for how both coordinates can refine the same informative representation. The implemented methods learn one correction shared across states: exact matching across all states additionally requires their mapped increments $\Delta u_j$ to agree, which a nonlinear $U_\theta$ does not guarantee.

Dear reviewers,

Please be aware that little time remains for the author-reviewer discussion. As the deadline is approaching, we kindly request that you refrain from asking for additional experiments. Please summarize your final position concisely.

Sincerely, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.