acceptodds
Under review as a conference paper at ICLR 2027

Off-Manifold Collapse in Guided Protein Language Models

Abstract

Protein language models are widely used priors for protein sequence design, and a growing body of work controls them at inference time as an alternative to fine-tuning. Such guidance faces a dilemma: mild enough to preserve natural activation statistics, it barely moves the property; strong enough to move it, the generations become progressively harder to fold. We show the failure has a specific and cheaply detectable signature, an off-manifold collapse of the model's own representations. Guided activations fall toward a region statistically indistinguishable from random amino-acid input, and the sequences degenerate to low complexity, yet the property oracle being optimized can still score these generations as a success. The optimized oracle can therefore fail to witness the collapse and, for solubility, can actively reward it, whereas structure and composition expose the failure. Because the failure is already visible in a finished candidate, we detect it at the output rather than modify the generator. We introduce a cheap density prior over natural protein activations and keep only the candidates that remain typical under it, a training-free post-hoc step we call Mahalanobis filtering. At matched guidance settings it improves both the property score and the structural plausibility of the sequences it keeps at negligible cost, without touching the generator, and transfers across different guidance method.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
protein language models, inference-time guidance, activation steering, protein sequence design, out-of-distribution detection, reward hacking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_u1GaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper concerns the problem of steering a protein language model. The challenge is that current approaches for steering language models cannot reliably produce coherent sequences that also satisfy the factor that is being steered. Overall the paper used a filtering approach (based on Mahalanobis distance) for identifying structurally plausible sequences.

Critical strengths
  • The paper demonstrates a very concrete failure that is induced by guidance and steering. Of course, prior work showed this too. However, here this paper finds the representation signature: i.e, that guiding activations leads to activations for random amino acids. They show the failure applies to multiple guidance methods.
  • They show that optimizing a property predictor can reward degenerate sequences.
  • Their detection is very simple, and might be more commonly adopted in the future.
Critical weaknesses

The novelty of work is quite limited. I outline how I see it:

  • Activation steering is already well known, the difficult of steering coherently is also well known, filtering is also well known. In fact, there are previous works on sparse autoencoders and proteins, and steering protein language models.
  • It would be important to see whether activation-based filtering matches or outperforms their proposed approach.
Questions for authors
  • Frey et al. (ICLR 2024), Protein Discovery with Discrete Walk-Jump Sampling, used a distributional conformity for evaluating generated proteins. How does the proposed Mahalanobis filtering compare with the Distributional Conformity Score as a measure of biological plausibility? In particular, does activation-space typicality detect implausible guided generations that sequence-level distributional conformity measures fail to identify?
2 comments

We thank the reviewer for recognizing the representation level failure and the possible utility of a simple detector. We agree that activation steering and post-hoc selection are established techniques, and will make the related protein steering and representation analysis literature more explicit.

W1–W2. Novelty and comparison with alternative activation-based methods

We thank the reviewer for highlighting prior work on protein activation steering and sparse autoencoders which provide valuable insights. Our contribution is not a new steering algorithm or the general idea of filtering, but the identification and characterization of a specific failure mode in guided protein generation.

As shown in Fig. 3, stronger steering causes protein representations to contract toward the natural protein mean, approaching the radial regime of random amino acid sequences, even as the predicted target property improves. We further observe this inward contraction under predictor gradient guidance, indicating that it is not limited to vector steering.

This finding motivates our Mahalanobis filter: a simple, training-free diagnostic specifically designed to detect inward representation collapse.

To test whether other activation-based methods provide similar filtering performance, we compared alternative representation scores on the same 3,454 generated sequences at matched retention rates.

Table 1. Comparison with alternative activation based filtering methods.

Filtering methodpLDDT @ 20% retained ↑pLDDT @ 50% retained ↑
Mahalanobis (ours)63.053.1
SAE reconstruction error57.953.2
GLP typicality57.353.3
L2 distance to mean54.151.4
kNN activation distance53.749.0
Random46.946.9

M achieves better selection quality than the tested simple activation distance baselines and performs competitively with the learned GLP prior. At 20% retention, M improves mean pLDDT over GLP typicality (63.0 vs. 57.3), while at 50% retention their performance is nearly identical.

Unlike GLP, our method requires no additional model training and uses only precomputed activation statistics.

SAE reconstruction error performs comparably at 50% retention (53.2 vs. 53.1), but M performs better at 20% retention (63.0 vs. 57.9) without requiring auxiliary model training.

Together, these results support our central contribution: a specific representation level failure signature that can be detected efficiently and used to improve the selection of guided protein sequences.

Q1. Comparison with Distributional Conformity Score (DCS)

We thank the reviewer for pointing out Frey et al. (ICLR 2024). DCS measures conformity to a reference distribution through the joint density of protein properties.

We evaluated DCS at matched retention rates using UniRef50 reference sequences and the released conformity score procedure. To ensure a meaningful comparison, we tested both a physicochemical feature implementation and an Appendix F style adaptation using the three transferable features from the original paper. We further strengthened DCS by incorporating PLM pseudo perplexity into the joint density, as permitted by its original formulation.

Table 2. Structure confidence selection using Mahalanobis and DCS.

Selection methodESMFold pLDDT ↑
Mahalanobis (ours)63.0
DCS (27 sequence features)48.2
DCS (Appendix F-style + PLM feature)51.7
Random47.2

M outperforms even the strengthened DCS variant at 20% retention. The two methods also identify substantially different candidates: for the 27 feature DCS adaptation, their top 20% selections overlap by only 28.7%. Candidates selected exclusively by M achieve higher ESMFold pLDDT than those selected exclusively by DCS (61.2 vs. 45.1).

Table 3. Structural confidence of candidates selected by only one method.

Candidate groupNumber of sequencesESMFold pLDDT ↑
Selected by M only51961.2
Selected by DCS only51945.1
Selected by both20965.3
Selected by neither2,33744.1

Dear Reviewer,

We would appreciate any feedback on our rebuttal and whether it addresses your concerns. We would be happy to clarify any remaining questions.

Thank you for your time and consideration.

Reviewer_DC44OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies the phenomenon that property guided protein design faces degradation of validity of generated protein sequences. Particularly, strong property guiding signals often result in sequences that are hard to fold and off-manifold in the natural protein sequence space. Experiments over different guiding methods and properties established the ubiquitousness of this problem. To tackle this issue, the authors propose a filtering method based on the Mahalanobis radius, which defines the deviation of the protein language model activations from the statistical mean. Experiments show that with proper filtering threshold of the radius, improvement in both target property and sequence quality are observed, with a sacrifice in generation acceptance rates.

Critical strengths
  1. The paper proposes a simple yet clean metric that can be used to filter low-quality sequences in property guided protein design. The value of Mahalanobis radius in depicting the degree of off-manifold in generated sequence set is established.

  2. Efficiency of the filtering method is discussed and other potential alternative is discussed and compared (the GLP flow NLL).

  3. The paper is clearly written and easy to follow.

Critical weaknesses
  1. Filtering low-quality sequence after generation is a simple idea to improve the overall folding probability of designed protein sequences. However, there can be tremendous different filtering metrics to use. Although the paper provides some analysis of correlation between Mahalanobis radius and GLP flow NLL, which represents the probability in the latent space of natural sequences, the fundamentality of the radius in depicting the off-manifold is not well established.

  2. Though the approach is simple, it lacks clear discussion in potentially more elaborated choices to make the filtering more effective. For example, the selection of threshold seems somewhat arbitrary.

  3. Comparison with potential choices other than the Mahalanobis radius is weak. For example, the GLP flow NLL could itself be a metric. It could provide better acceptance rate, though with higher cost. The pLDDT or effective alternatives of pLDDT can also be valid choices for the filtering. The necessity of using the Mahalanobis radius is weakly argued. The issues of low acceptance rate and how to make a better trade-off is not discussed either.

  4. Besides filtering, there could be more reasonable approaches to trade-off between target properties and sequence quality in the steering process itself. These methods are not well studied and compare in this paper.

Questions for authors
  1. In Section 3.4, the authors claim that the collapse appears under predicted-gradient guidance. Is there any evidence to support this?

  2. Please see the weakness section for other questions.

3 comments

We thank the reviewer for recognizing the simplicity and efficiency of our method. Our contribution is not simply filtering guided candidates, but identifying steering induced inward collapse in protein representations and developing an efficient detector targeted at this failure. We address the metric choice, alternative selectors, threshold sensitivity, and generator level interventions below.

W1 / W3. Why Mahalanobis?

Fig. 3 shows that stronger steering contracts ESM-2 representations toward the natural protein mean, approaching the radius of random amino acid inputs (~600 vs. 604). This motivates detecting abnormally small Mahalanobis (M) radii. The original manuscript compares M with a learned activation prior (Sec. 5.3) and reports computational cost (Appendix F). We now evaluate GLP and other selectors on the same 3,454 generated candidates. Each selector ranks sequences within the same generation pool, and the top 20% or 50% are evaluated using ESMFold pLDDT.

Table 1. Structural confidence at matched retention.

Selection methodpLDDT @ 20% ↑pLDDT @ 50% ↑Compute ↓
Mahalanobis (ours)63.053.11×
GLP NLL (density)37.541.225.7×
GLP NLL (typicality)57.353.325.7×
Pseudo-PPL (650M)63.054.410.2×
L2 distance to mean54.151.41×
Random46.946.9—

GLP likelihood ranks candidates poorly, while GLP typicality performs better. M achieves higher pLDDT than GLP typicality at 20% retention (63.0 vs. 57.3) and similar results at 50%. Pseudo-perplexity is competitive, matching M at 20% retention, but requires 10.2–32.6 times more scoring time. These results support M as an efficient detector of inward collapse, not a universal protein quality ranker.

We also tested the suggested folding predictor baseline: candidates are ranked using ESMFold and evaluated independently with OmegaFold to avoid using the same predictor for selection and evaluation.

Table 1b. Folding predictor selection at matched retention.

Selection methodOmegaFold @ 20% ↑OmegaFold @ 50% ↑Compute ↓
Mahalanobis (ours)64.757.21×
ESMFold pLDDT ranking68.059.52.3×
Random51.951.7—

ESMFold ranking yields higher OmegaFold confidence, whereas M directly measures the representation contraction identified in our study, at lower cost. Our objective is to detect steering induced inward collapse in PLM representation space, not to maximize pLDDT. Structural confidence and representation contraction measure different aspects of the generated sequences, making folding predictor ranking an informative but distinct comparison.

W2 / W3. Threshold choice and acceptance

Appendix E (Table 8) already examines threshold sensitivity. We further evaluated thresholds defined by natural protein percentiles.

Table 2. Quality and retention across thresholds.

ThresholdRetainedESMFold gain
Natural p1085.6%+2 pts
Natural p2550.7%+8 pts
Original k=128.8%+12 pts
Natural p5023.9%+14 pts

The improvement persists across thresholds, so it does not depend on k=1. More permissive thresholds retain more sequences but yield smaller confidence gains. We will clarify how the reference percentile thresholds relate to the original k=1 criterion and discuss the cost of lower acceptance rates. The threshold is therefore a tunable quality versus retention tradeoff, not a fragile optimum.

W4. Why not improve generation directly?

We already tested generator interventions in Appendix G (Table 10): 32 combinations of vector changes, bounded steering, layer scopes, and strengths. For thermostability, bounded steering improves predicted Tm from 55.3°C to 64.1°C while maintaining pLDDT (70.7 to 71.3). Comparable gains were not observed in the tested solubility settings. These results demonstrate that direct repair can succeed, but its effectiveness depends on the target property and intervention design. In contrast, M screens completed sequences without modifying the controller and can be applied across different generation strategies.

Q1. Collapse under predictor gradient guidance

This result is already reported in Secs. 3.4 and 5.2 (Table 2), and Appendix D (Table 7). Predictor gradient guidance modifies hidden activations using the property predictor's gradient, without a fixed steering vector. In the lyso/therm setting, increasing η from 0.05 to 0.20 decreases M from 1039 to 769 and ESMFold pLDDT from 55 to 41. This establishes that the observed inward contraction is not restricted to fixed vector steering, directly addressing the reviewer's question.

Thanks for the additional results and answers to the questions. The experiments show that Mahalanobis can be used as an efficient filtering metric, compared with other alternatives. However, it would be valuable to show the advantage of Mahalanobis in terms of Pareto frontier of plotting acceptance rate and structure quality. Acceleration in filtering alone is insufficient. The quality and success rate when designing the protein is also important. Faster filtering, with lower acceptance can still result in slower design throughput.

We thank the reviewer for the thoughtful follow up.

Regarding the quality acceptance trade off, our rebuttal already provides retention comparisons at 20% and 50% (Tables 1 and 1b), alongside four threshold settings covering acceptance rates from 23.9% to 85.6% (Table 2). These results demonstrate competitive structural quality at substantially lower scoring cost. While a more detailed Pareto curve could provide further insights, we hope these results address the main concern.

We would also like to clarify that our primary contributions are to characterize off manifold representation collapse under inference time guidance and develop an efficient, training free detector for this failure. We demonstrate this phenomenon under both activation steering and predictor gradient guidance, and show that the same detector transfers across these mechanisms without recalibration. Our focus is on detecting failures in generated candidates, rather than optimizing the generation process itself.

We appreciate the reviewer's suggestions and consideration.

Reviewer_KXwVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper studies how inference-time guidance affects protein sequences generated by ESM-2 650M for predicted solubility and thermostability. Under both activation steering and predictor-gradient guidance, the authors observe that stronger guidance can shrink the activation radii of completed sequences toward the range seen for random amino-acid sequences, alongside lower ESMFold pLDDT. They propose a post-hoc filter that rejects candidates with small Mahalanobis radii relative to a UniRef50 reference. Within fixed candidate pools, the retained subsets have higher mean ESMFold pLDDT, with retention varying across guidance settings.

Critical strengths

Building on Huang et al. (ICML 2025), who reported degradation under strong steering, this paper identifies a more specific signature in ESM-2 650M: the activation radii of completed sequences shrink toward the range observed for random sequences. A simple post-hoc filter enriches for higher ESMFold confidence within fixed candidate pools, and the authors report the fraction retained alongside the gain.

Critical weaknesses
  1. Structural validation relies on one predictor. ESMFold uses ESM-2 3B, while the filter uses ESM-2 650M. Mean pLDDT measures confidence in predicted local geometry and provides only indirect evidence of structural plausibility. Rescoring the same candidates with ESMFold2 in single-sequence mode would test whether the enrichment holds across predictors.
  2. The fixed-pool experiments lack matched-retention baselines. Comparing retained candidates with the full pool shows enrichment. Pseudo-perplexity is compared only on runtime, leaving open whether it or a simple sequence-complexity filter would achieve similar enrichment at the same retention rate.
  3. Property gains lack independent validation. The same predictors guide generation and score the resulting sequences, and their accuracy on strongly steered candidates is unknown. The thermostability predictor was trained without sequence redundancy reduction, which may make its held-out performance optimistic.
  4. Overlap with pretraining is unquantified. The Mahalanobis reference statistics are estimated from UniRef50, which also underlies ESM-2 pretraining. The manuscript does not quantify sequence overlap or test whether the cutoff and enrichment hold with a reference set separated from pretraining by sequence homology.
  5. Backbone generalization remains untested. Both guidance methods are evaluated on ESM-2 650M. Repeating the steering sweep on ESM-C with backbone-specific reference statistics would test whether the inward contraction and filtering gains generalize.
Questions for authors
  1. Using the same property predictor and starting sequences, can you backpropagate through ESM-2 to optimize amino-acid logits directly? At comparable predicted property gains and mutation rates, do the resulting discrete sequences show the same inward contraction in Mahalanobis radius?
  2. Does the post-filtering gain in mean pLDDT persist when the same candidates are rescored with ESMFold2 in single-sequence mode? Within each fixed candidate pool, how does Mahalanobis filtering compare with pseudo-perplexity and a simple sequence-complexity filter at matched retention rates?
  3. Do the reported gains in solubility and thermostability persist when the same candidates are scored by independent property predictors evaluated on homology-separated test sets?
3 comments

Thank you for recognizing the representation level failure. We conducted further validation, baseline comparisons, and optimization controls.

W1 / Q2. Independent structural validation

We rescored the same generated sequences with OmegaFold and ESMFold2, whose backbones differ from the ESM-2 representation used by M.

Table 1. Structural confidence across independent folding predictors.

PredictorUnguided ↑Guided ↑Guided + M ↑
ESMFold v167.243.456.1
OmegaFold73.849.960.5
ESMFold2 (ESM-C)79.557.165.7

All three predictors confirm that guidance reduces structural confidence and M filtering improves it on the same generated sequences, limiting the concern that the result is specific to ESMFold v1. Using ESMFold2, M also exceeds random selection in 22/28 settings on the retention quality summary (median advantage 0.0689). These are computational, not experimental, structural assessments.

W2 / Q2. Matched retention baselines

Sec. 5.3 and Appendix F compare M with a learned prior and report compute. We additionally evaluated 20 selectors on 3,454 candidates, ranking within each generation pool and retaining the top 20% or 50% for ESMFold evaluation.

Table 2. Structural confidence at matched retention rates.

SelectorpLDDT @ 20% ↑pLDDT @ 50% ↑Compute ↓
M (ours)63.053.11×
Pseudo-PPL (650M)63.054.410.2×
Pseudo-PPL (3B)62.954.532.6×
Sequence entropy48.346.5—
L2 to reference mean54.151.41×
Random46.946.9—

M matches pseudo-perplexity at 20% retention and remains competitive at 50%, with approximately 10–33× less scoring time. Unlike the original full pool comparison, this evaluation controls for retention. M targets inward contraction rather than general protein quality.

W3 / Q3. Independent property validation

We additionally evaluated thermostability with TemStaPro, which is independent of the predictor used for guidance. The table reports one matched setting.

Table 3. Independent thermostability evaluation.

MeasureUnguidedGuidedGuided + M
Optimization oracle: Tm (°C)51.965.868.9
TemStaPro score (0–1)0.3510.6460.623
ESMFold pLDDT63.648.063.1

Guidance raises independent predicted thermostability (0.351 to 0.646). Filtering recovers pLDDT (48.0 to 63.1) while retaining most of the thermostability gain (0.623). This independent predictor offers a check against relying only on the optimization oracle.

W4. Reference and pretraining overlap

M measures activation typicality, not sequence novelty. UniRef50 estimates the natural representation distribution, not experimental quality labels. Its similarity to the pretrained PLM distribution is consistent with this objective. We will clarify this distinction; exact pretraining sequence overlap has not been quantified.

W5. Generalization across PLM backbones

We repeated the steering study using ESM-C 600M as the generator and recomputed backbone specific M statistics (128 sequences per setting).

Table 4. Inward contraction under ESM-C steering.

Steering αΔTm (°C) ↑ΔMpLDDT before → after M ↑
0.25+2.6−4161.9 → 69.6
0.5+0.6−21847.0 → 63.2

At α = 0.25, steering raises predicted Tm (p = 8.0 × 10⁻⁶) while reducing M. Filtering improves pLDDT to 69.6, above the unguided baseline (65.2). At α = 0.5, contraction strengthens and filtering again improves pLDDT. Both effects extend beyond ESM-2.

Q1. Does direct amino acid optimization cause the same contraction?

The manuscript already reports contraction under vector steering and predictor gradient guidance (Secs. 3.4, 5.2; Appendix D). We added HotFlip style optimization and random substitution controls starting from 42 matched unguided sequences.

Table 5. Optimization controls at comparable mutation rates.

MethodAA changedΔTm (°C) ↑ΔM ↑ΔpLDDT ↑
Activation steering (α = 0.5)19.4%+13.2−458−30.2
Gradient guidance (η = 0.2)19.7%+27.9−413−33.3
HotFlip style optimization20.7%+39.2−147−7.0
Random substitutions20.7%−3.8−74−10.2

With similar mutation rates, HotFlip achieves larger predicted Tm gains but much less contraction (−147 vs. −413/−458) and pLDDT loss (7 vs. 30–33 points). Severe contraction is not inevitable when optimizing a property predictor; it depends on the method. Our contribution is to characterize and detect this failure.

Thank you for the additional experiments.

  1. The HotFlip results address my concern about whether severe contraction depends on the optimization method. A direct comparison with BindCraft-style continuous amino acid logit optimization remains open.
  2. The independent thermostability results are encouraging, and I think a similar trend is likely for solubility, which still needs independent validation.

Overall, I find the new evidence convincing enough to raise my rating to 3. ; ) I hope the revised manuscript incorporates these results and addresses the remaining points in my review.

Thank you for your constructive feedback and for reconsidering your assessment! We are glad that the additional experiments helped address your main concerns.

We will incorporate these results into the revised manuscript and explore more with continuous-logit optimization, independent solubility validation, and pretraining overlap.

We appreciate your thoughtful suggestions, which have helped strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.