acceptodds
Under review as a conference paper at ICLR 2027

PLM-CARE: Biosafety Assessment and Quality-Matched Preference Alignment for Protein Language Models

Abstract

Protein language models (PLMs) enable increasingly controllable protein design, yet it remains unclear whether different PLM architectures produce outputs with predicted virulence-factor (VF) association, and whether such signals can be reduced without degrading prespecified sequence-quality proxies. We introduce PLM-CARE, a cross-architecture framework that generates candidates from ESM3 and PoET-2 through their native masked and causal paths, converts VirulentHunter-scored candidates into quality-matched preferences, and fine-tunes each model with a shared reference-relative preference objective instantiated through architecture-specific scoring. Under matched native regeneration on held-out, risk-enriched contexts, preference fine-tuning lowered the mean DTVF probability (R_DTVF) from 0.67 to 0.52 for ESM3 and from 0.88 to 0.74 for PoET-2; a second external classifier, PSSM-based VirulentPred 2.0, showed mean paired prompt-level reductions in predicted-positive fraction (R_VP2) of 17.5 and 18.9 percentage points, respectively, on its supported-input subsets. Both external classifier endpoints decreased across three training runs per architecture. Prespecified quality proxies showed no observed degradation for either architecture. Together, these results indicate that computationally predicted VF-associated signals in outputs from the evaluated masked and causal PLMs under these contexts can be reduced by architecture-adapted fine-tuning. The code is available at https://anonymous.4open.science/r/plm-care-review-B2DF/.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Computational Biosecurity Risk Assessment, Protein Language Models, Preference-based Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8chYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PLM-CARE, a framework for assessing and reducing predicted virulence-factor (VF) association in protein sequences generated by protein language models. The framework combines evolutionary-context generation, multiple VF predictors, and quality-matched preference alignment. VirulentHunter is used to construct preference pairs, while a DPO-based objective is adapted to the masked generation of ESM3 and the autoregressive generation of PoET-2.

Experiments on held-out evolutionary contexts show reductions in VF-associated predictions from two external classifiers, DTVF and VirulentPred 2.0, after fine-tuning. The authors also evaluate sequence validity, evolutionary-profile compatibility, and other sequence-quality proxies, reporting no observed degradation in these metrics.

Critical strengths

1.The paper evaluates both ESM3 and PoET-2, covering masked and autoregressive generation. The consistent reductions in predicted VF association across the two models suggest that the proposed alignment approach is not limited to a single generation architecture.

2.The authors use separate classifiers for preference construction and evaluation, report results across multiple training seeds, and examine several sequence-quality metrics. The homology-aware data splitting and ablation experiments also strengthen the experimental analysis.

Critical weaknesses
  1. The proposed framework mainly combines existing VF predictors, preference filtering, and DPO-based fine-tuning. While applying the framework to different PLM architectures is useful, the methodological contribution appears relatively limited. It is also unclear whether the proposed quality-matched preference construction offers substantial benefits over standard preference optimization.

  2. The evaluation relies entirely on predicted VF-associated scores, without direct biological validation. Although multiple classifiers are considered, their agreement at the prompt level is relatively weak. This raises concerns about whether the observed reductions reflect meaningful changes in biological properties or simply differences in the sequence features captured by these predictors.

  3. The experiments focus on two PLMs and a relatively narrow set of risk-enriched evolutionary contexts. It remains unclear whether the method generalizes to other protein families or generation settings. In addition, the reported quality metrics mainly assess sequence validity and evolutionary compatibility, which do not necessarily indicate preserved biological function.

Questions for authors
  1. Section 4.5 includes an ablation that removes the quality-matching constraint, but the reported results mainly focus on sequence-quality metrics. Could the authors also report the corresponding changes in VF-associated predictions for this ablation? This would help clarify whether quality matching provides a better trade-off between reducing VF-associated signals and preserving sequence quality. Comparisons with simpler fine-tuning baselines would further help establish the contribution of QMPA.

  2. The paper reports consistent overall reductions across external classifiers, but the prompt-level effect correlations are relatively weak, particularly for PoET-2. Could the authors further analyze these disagreements and discuss whether the alignment primarily optimizes features associated with the preference-labeling classifier rather than producing consistent changes in biologically meaningful VF-associated properties?

  3. While the paper reports sequence uniqueness and embedding-space visualizations, these analyses do not fully characterize the diversity or structural quality of generated proteins. Could the authors provide additional evaluations, such as pairwise sequence identity and predicted structural confidence? It would also be helpful to assess whether fine-tuning preserves the predicted functional properties of generated sequences.

  4. Both models are evaluated primarily on risk-enriched evolutionary contexts derived from VF-associated proteins. How well does the alignment generalize to other protein families or generation conditions? In particular, does the method affect the generation quality or functional predictions of proteins that are not associated with virulence factors?

9 comments
AuthorsOpenReview ↗

Response to Reviewer 8chY — Q1 (1/2)

Thank you for your constructive feedback and for highlighting the strengths of our cross-architecture evaluation.

The original quality-matching ablation focused on the direct objective of the constraint. It tested whether introducing quality matching during preference optimization helps preserve the quality proxy. We therefore first reported the change in Q_profile for QMPA relative to noQ. Your suggestion makes the comparison more complete and helps us assess how quality matching relates to improvements in VF-associated scores.

We therefore expanded the quality-matching ablation. First, we evaluated Base, QMPA, and noQ on a common cohort to make the ablation more rigorous. In addition to changes in Q_profile, we compared predictions for Base, QMPA, and noQ from the external evaluators DTVF and VirulentPred 2.0 (VP2). ESM3 VP2 is reported on the 129 contexts scored in all three arms.

Quality-matching ablation on a common cohort

ArchitectureMethodQ_profileΔQ_profile vs. BaseDTVFVP2 positive rateVH
ESM3Base47.4692 [45.5678, 49.7408]00.666 [0.556, 0.765]74.3% [56.1%, 89.9%]0.604 [0.560, 0.654]
ESM3QMPA48.4256 [46.0563, 51.3729]+0.95640.517 [0.367, 0.666]56.9% [41.7%, 76.7%]0.377 [0.289, 0.457]
ESM3noQ47.0077 [44.6992, 49.8306]−0.46150.614 [0.500, 0.727]62.6% [47.2%, 81.8%]0.466 [0.413, 0.518]
PoET-2Base29.4934 [28.1766, 30.8825]00.876 [0.843, 0.904]82.8% [79.5%, 85.7%]0.553 [0.498, 0.605]
PoET-2QMPA40.0968 [37.5367, 42.3906]+10.60340.735 [0.668, 0.801]61.8% [55.2%, 68.0%]0.408 [0.361, 0.463]
PoET-2noQ33.3719 [31.9203, 34.8152]+3.87850.900 [0.841, 0.942]72.5% [64.4%, 79.7%]0.290 [0.229, 0.369]

Comparing Q_profile for QMPA and noQ on the same cohort for ESM3 and PoET-2 yields the same conclusion as the original manuscript. Removing the quality-matching constraint lowers the fine-tuned models' generation quality as measured by Q_profile relative to QMPA.

To answer your follow-up question, we first scored all three arms with the two external evaluators. For ESM3, both evaluators gave QMPA lower mean risk scores than noQ, with the same ordering of means in both cases (QMPA < noQ < Base). The pattern was different for PoET-2. QMPA again received lower risk scores, while noQ's DTVF score increased relative to Base. This was unexpected, so we also computed VirulentHunter (VH) scores for all three PoET-2 arms. We found that noQ achieved the greatest improvement on VirulentHunter, the classifier used to construct the preferences, but that advantage did not carry over to the external evaluators. This suggests that removing the quality-matching constraint may lead fine-tuning to cater to the classifier's preferences at the expense of learning genuine biological properties. We address this point further in our response to Q2.

We likewise computed the classifier scores for all three ESM3 methods. The VirulentHunter scores followed the same trend as the external evaluator scores.

The additional risk scores show why VirulentHunter, the classifier used during training, is insufficient on its own to judge the effect of fine-tuning. They also show that MVA's separation of the labeling classifier from the external evaluators gives a fuller picture of where fine-tuning has an effect. Finally, they show that QMPA can retain an improvement on VirulentHunter while transferring that improvement to external evaluators, yielding more consistent improvements in the average external scores.

AuthorsOpenReview ↗

Response to Reviewer 8chY — Q1 (2/2)

In addition to the fine-tuning baseline without the quality-matching constraint, we added a supplementary chosen-only SFT experiment as a simpler baseline. This addresses the concern in your critical weaknesses that the proposed quality-matched preference construction may offer limited advantages over standard preference optimization. The results show that QMPA also has advantages over chosen-only SFT. We trained SFT using only the preferred sequence from each QMPA pair and evaluated both methods on the same held-out contexts. The table below presents the results.

Comparison of QMPA and chosen-only SFT

ArchitectureMethodMean DTVFΔDTVF vs. Base (95% CI)Mean Q_profileΔQ_profile vs. Base (95% CI where available)
ESM3QMPA0.5166−0.1491 [−0.2510, −0.0731]48.4256+0.9564 [0.2455, 1.7116]
ESM3Chosen-only SFT0.6478−0.0179 [−0.0353, −0.0052]47.1185−0.3507 [−0.5221, −0.2129]
ESM3Base0.6656047.46920
PoET-2QMPA0.7351−0.1405 [−0.1819, −0.0955]40.0968+10.6034 [9.3642, 11.8480]
PoET-2Chosen-only SFT0.9540+0.0784 [0.0673, 0.0900]22.9128−6.5806
PoET-2Base0.8756029.49340

Direct comparison of QMPA and chosen-only SFT

ArchitectureΔDTVFΔQ_profileΔVirulentHunter
ESM3−0.1312 [−0.2173, −0.0657]+1.3071 [0.6155, 2.1325]−0.1894 [−0.2406, −0.1450]
PoET-2−0.2189 [−0.2424, −0.1965]+17.1840 [15.6148, 18.7695]+0.0564

For PoET-2, QMPA changed DTVF by −0.2189 and Q_profile by +17.1840 relative to SFT. The corresponding ESM3 analysis also favored QMPA on DTVF (−0.1312) and Q_profile (+1.3071).

These supplementary experiments demonstrate advantages for QMPA over baseline preference optimization.

In Q1, we added both a joint analysis of the quality-matching ablation and VF prediction scores and a chosen-only SFT control. We will replace the original quality ablation in Section 4.5 with the new quality analysis. We will also report the scores from the external evaluators and the preference-labeling classifier to extend the conclusions of the ablation. The chosen-only SFT control demonstrates QMPA's added value over simple fine-tuning, and we will add it to the ablation experiments in Section 4.5.

AuthorsOpenReview ↗

Response to Reviewer 8chY — Q2

We acknowledge the difference between the aggregate trends and the per-context results. To analyze these discrepancies, we conducted a supplementary paired analysis of cross-evaluator agreement on common contexts. For each of the 137 PoET-2 contexts that both DTVF and VP2 could score, we computed the change from Base to QMPA. On the 137 shared PoET-2 contexts, both external evaluators showed mean decreases after QMPA. Agreement on which individual contexts improved was only slightly above that expected from the marginal decrease rates (77.4% observed vs. 76.0% expected). These results support an aggregate effect across the two external predictors,We then compared how closely the classifiers agreed on the direction of the changes and on their ranking by magnitude. The table below presents the results.

Agreement between external evaluators on common PoET-2 contexts

MeasureResult
Mean DTVF change−0.1722
Mean VP2 positive-rate change−18.89 percentage points
Both endpoints decreased103/137
Only one endpoint decreased31/137
Neither endpoint decreased3/137
Observed agreement in direction77.4%
Expected agreement from marginal decrease rates76.0%
Cohen's κ0.056
Spearman's ρ for the changes (95% CI)0.023 [−0.264, 0.305]

The results show that DTVF fell by 0.1722 on average, the VP2 positive rate fell by 18.89 percentage points on average, and both measures fell in 103 contexts. Each measure fell in most contexts, so the expected agreement based on their marginal decrease rates alone was 76.0%, close to the observed agreement of 77.4%. The corresponding Cohen's κ was 0.056. The Spearman correlation for the magnitude of the changes was 0.023, showing little agreement in the rankings. Thus, both classifiers showed an average decrease across these contexts, while agreement beyond chance on which contexts improved was weak, as was agreement on which contexts improved the most.

To understand this pattern, we examined the prediction tasks and modeling approaches of the two evaluators in more detail. Both DTVF and VP2 are used to identify virulence-associated proteins. DTVF combines representations from the ProtT5 protein language model with a deep learning model to make predictions, while VP2 uses position-specific scoring matrices (PSSMs) as features for classification. Their prediction objectives are similar, but they extract and use different sequence information and may therefore respond differently to changes in the same set of sequences.

In addition, this analysis compared changes in continuous DTVF scores with changes in VP2 positive rates. A continuous score can change while a sequence stays on the same side of the positive classification threshold. Changes in the positive rate primarily reflect the fraction of samples crossing that threshold. The two statistics may therefore rank the magnitudes of change differently. The differences in models and statistics provide plausible explanations for the low correlation, although this agreement analysis leaves the contribution of each factor unresolved.

You asked whether our fine-tuning method simply teaches the model to generate “sequences that this classifier will score lower,” and whether these changes correspond to real, biologically meaningful reductions in virulence. Our evaluation was designed to address precisely this question. We use VirulentHunter to construct the preferred and rejected sequences, y+y^+ and y−y^-, and use DTVF and VP2 for evaluation, providing an analysis based on biological predictors. We use different predictors during training and testing. VirulentHunter is used only during training, and DTVF and VP2 are used only during testing. The test results show that the improvements obtained with VirulentHunter-labeled preference pairs transfer to the external evaluators. VirulentHunter is a leading protein virulence-factor predictor with a strong design. Using it as an optimization surrogate therefore helped the model learn biologically relevant preferences efficiently.

The supplementary experiment in this response clarifies what we mean when we say that the external evaluators corroborate each other. We will add the experiment and its main conclusions to Section 4.2, expanding the original comparison between evaluators.

Reviewer_5F4FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether protein language models trained on natural sequences produce outputs that automated classifiers associate with virulence factors, and whether preference fine-tuning can lower that score without degrading sequence-quality proxies. It frames this as a gap in existing work: prior red-teaming measures whether a model can be steered toward recovering a specified target, which does not characterize what a model produces under ordinary conditional generation.

PLM-CARE has three parts. Generation draws candidates from two architectures with different factorizations — ESM3 (masked) and PoET-2 (causal) — conditioned on evolutionary context, with no risk score used to guide decoding. Assessment applies three published classifiers, with a separation of roles: one supplies the training preference labels, and two others serve only as post-hoc evaluation endpoints and contribute no training signal. Alignment constructs preference pairs matched on a profile-compatibility score, so that a pair differs in predicted risk but not in family fit, and optimizes a DPO-form objective through a shared reference-relative margin instantiated separately for the masked and causal scoring rules.

Evaluation regenerates from 200 held-out contexts, 16 draws per context (3,200 per policy), retaining every attempted draw, with three training runs per architecture. Uncertainty uses source prompts as statistical units and homology components as bootstrap clusters (10,000 resamples), without pooling across seeds. The primary external endpoint falls from 0.666 to 0.517 for ESM3 (paired change −0.149) and 0.875 to 0.735 for PoET-2 (−0.140); a second external endpoint falls 17.4 and 18.9 percentage points on its eligible subsets; the quality proxy rises by +0.97 and +10.61. The authors conclude that predicted risk-associated signal can be reduced in these contexts, and state explicitly that this does not establish biological safety.

Critical strengths

The measurement protocol is designed against the obvious ways this result could be manufactured, and the paper is unusually disciplined about it (§3.2, §4.1, App. A.4). The risk scores are applied only after generation, decoding is never guided by them, every attempted draw is retained with no reranking or replacement sampling, and the classifier that supplies training preferences is kept distinct from the two used to evaluate. The statistics match the dependence structure of the data rather than treating sequences as independent: source prompts are the units, draws are summarized within prompt, homology components are the resampling clusters, and seeds are not pooled. Reproducibility is handled at the same level, with pinned tool versions, checkpoint fingerprints, and a recorded scoring configuration that reproduces the probabilities exactly. For a claim of the form "our intervention lowers a score we also chose", this is close to the right set of precautions, and it is more careful than much of the surrounding literature.

The paper states the limits of its own evidence rather than leaving them to the reader (§3.3, §4.2, §5, Ethics Statement). It says directly that evaluator training-record independence is unverified and therefore declines to treat endpoint agreement as a safety certificate; it reports that for PoET-2 the two external endpoints agree on the direction of the aggregate change but correlate weakly at the level of individual prompts; it describes chosen and rejected sequences as relative preferences rather than verified safe and unsafe proteins; and it frames the conclusion around predicted, computational endpoints in selected contexts. The dual-use discussion is substantive rather than a formality. This candor is what makes the paper's numbers usable as evidence, and it should be credited independently of whether the central claim survives.

Critical weaknesses

Three of the paper's own results point at a distribution-shift explanation for the headline effect, and the paper does not test it (§4.2, §4.3, §4.5, App. A.8). The concern is that fine-tuning may have moved generations toward a region these classifiers score lower, rather than having reduced a coherent underlying property. The paper's own evidence is consistent with that reading, in three places that it reports separately but never connects:

  • The quality proxy rises substantially for PoET-2 (+10.61, versus +0.97 for ESM3) at the same time as the risk score falls. A large simultaneous move in both suggests the policy has shifted toward a more canonical region of sequence space, which would be expected to lower a classifier score for reasons unrelated to the intended mechanism.
  • In the context-randomization ablation, the gap between base and fine-tuned policies narrows from roughly 0.227 to 0.032 as conserved context is randomized. The paper notes this does not establish a context-independent effect size; read the other way, the effect largely disappears once the conditioning that defines the evaluated region is removed.
  • For PoET-2, prompt-level rank correlation between the two external endpoints is near zero (ρ = 0.023). If both tracked a shared property at the level of individual contexts, some agreement would be expected; near-zero correlation is what one sees when each classifier responds to its own features.

A reviewer cannot distinguish "reduced risk-associated signal" from "moved away from the classifiers' positive region" on the evidence given. The informative control is a negative one: apply the same pipeline with the preference direction reversed or with a risk-irrelevant preference matched on the same quality constraint, and show the endpoints do not move comparably. Also diagnostic would be classifier behavior on natural held-out sequences of known status under the same scoring configuration, which would indicate whether the endpoints are calibrated on inputs of this kind at all. This is the main thing standing between the paper's narrow claim and the significance it implies. It is addressable without new infrastructure, since the pipeline and evaluators are already in place.

The quality-matching constraint, one of the three named contributions, is not supported by a controlled comparison (§4.5, Figure 5a). The ablation reports that removing the constraint changes mean quality by −0.66 for ESM3 and +0.48 for PoET-2, "versus constraint-enabled gains of +0.97 and +10.61". But the ablation's base values (roughly 31.3 and 19.7) differ substantially from the main experiment's (47.5 and 29.5), so the two sets of numbers are not measured on the same cohort and the comparison does not isolate the constraint. On PoET-2 the unconstrained variant also improves quality, which is the opposite of what the argument needs. Since quality matching is what distinguishes this preference construction from a straightforward risk-ordered one, and since preference accuracy on held-out pairs is modest (65.5% and 64.5%), the contribution needs a matched-cohort ablation reporting the same paired statistics and intervals used elsewhere in the paper.

Questions for authors
  1. Clarification. Is a negative-control arm available — the same pipeline with the preference direction reversed, or with a risk-irrelevant preference under the same quality constraint? Endpoints that move comparably there would indicate the shift is not specific to the intended property. This is the single result most likely to change my assessment.
  2. Clarification. In the quality-constraint ablation, why do the base values differ from the main experiment? Can the ablation be reported on the same cohort with paired changes and confidence intervals, so the constraint's contribution is isolated?
  3. Clarification. How do the two external classifiers score natural held-out sequences of known status under the identical scoring configuration? This would indicate whether their operating points are meaningful for inputs of this kind.
  4. Clarification. Given that the base/fine-tuned gap narrows sharply under context randomization, is the effect better described as specific to the evaluated contexts than as a property of the fine-tuned policy?
  5. Clarification. How should the near-zero prompt-level correlation between the two external endpoints for PoET-2 be interpreted, given that directional agreement in the aggregate is offered as corroboration?
  6. Suggestion. Relating the contribution more explicitly to the cited prior safety-alignment work on protein language models would help; at present the delta is left implicit.
9 comments
AuthorsOpenReview ↗

Response to Reviewer 5F4F — Q1

Thank you for your thoughtful assessment of our evaluation protocol and for raising these important questions.

To determine whether the direction of the preference pairs matters, we designed a negative control to test whether the scores would still decrease if we used the same training pipeline with reversed preferences or with preferences unrelated to risk.

The random-preference DPO control for PoET-2 used the same unordered quality-matched pairs, reference likelihoods, training seed, optimizer configuration, 600-step training budget, and fixed epoch-3 checkpoint rule. We randomized the direction of the pairs independently of the risk scores. Half of the pairs in each data split retained their original direction. Reference likelihoods stayed attached to their corresponding sequences. The mean signed VirulentHunter (VH) difference in the training set was +0.65087 with the original direction and −0.00075 with the randomized direction. We used the same decoding configuration and frozen scorers for evaluation, with 16 fresh draws per context. The common analysis cohort comprised 200 contexts from 66 components. VirulentPred 2.0 (VP2) covered 137 contexts from 45 components, with 2,192 draws per arm. We obtained the 95% confidence intervals from 10,000 paired bootstrap resamples of components. The table below compares QMPA, Random, and Base.

PoET-2 random-preference control and common evaluation setup

ContrastΔDTVF (95% CI)ΔQ_profile (95% CI)ΔVP2, percentage points (95% CI)
QMPA − Base−0.1405 [−0.1799, −0.0962]+10.6034 [8.3289, 13.0251]−19.39 [−24.40, −14.01]
Random − Base+0.0041 [−0.0041, 0.0128]+0.2738 [0.1536, 0.4135]−3.10 [−5.68, −0.70]
QMPA − Random−0.1447 [−0.1833, −0.1017]+10.3296 [8.0569, 12.7721]−16.29 [−20.01, −11.90]

The QMPA and Random arms for PoET-2 used the same unordered quality-matched pairs, reference likelihoods, training seed, and optimizer configuration. Both had a budget of 600 training steps and used the fixed epoch-3 checkpoint rule. The Random arm independently randomized pair direction, retaining the original direction for half of the pairs in each data split. Reference likelihoods remained tied to their corresponding sequences throughout. The mean signed VirulentHunter difference in the training set was +0.65087 for the original preference direction and −0.00075 for the randomized preference direction.

We used the same decoding configuration and frozen scorers for evaluation, with 16 fresh draws per context. The common analysis cohort comprised 200 contexts from 66 homology components. The VP2 subset comprised 137 contexts from 45 components, with 2,192 draws per arm. We obtained the 95% confidence intervals from 10,000 paired bootstrap resamples of homology components.

Randomizing the preference direction slightly decreased VP2 and slightly increased Q_profile. The direct comparison shows that, in this PoET-2 experiment, the intended preference direction produced larger changes in the external scores and in family compatibility. This result supports an effect of preference direction on PoET-2.

The random control also shows that the substantial quality gain for PoET-2 is associated with the preference direction. Both arms used the same quality-matched pairs and optimization procedure. Randomly flipping half of the preference labels reduced the Q_profile gain from +10.6034 to +0.2738. This led us to ask whether sequences with higher Q_profile scores had inadvertently become the preferred sequences, y+y^+, when we constructed the QMPA preference pairs. We therefore checked the average quality scores in the training pairs. The pattern went in the opposite direction from what we had hypothesized. The preferred sequences, y+y^+, scored 0.7693 points lower than the rejected sequences, y−y^-, on average. The evidence supports an effect of preference direction on PoET-2's generation quality.

The random-preference control tests whether the direction of the pairs contributes beyond the general effects of fine-tuning. The experiment also establishes a link between preference direction and generation quality, which is an additional finding of interest. We will add this experiment as a new part of the ablation in Section 4.5.

AuthorsOpenReview ↗

Response to Reviewer 5F4F — Q2

The baseline quality scores in the original quality-matching ablation differed from those in the main experiment because the original ablation used a separately constructed held-out cohort. Base therefore had different means on the two cohorts. This creates a problem because subtracting each cohort's own Base score can leave differences between the evaluation cohorts in the comparison. The fine-tuning effect may also vary by context. We therefore improved the quality-matching ablation by fixing the evaluation contexts.

In the supplementary ablation on a common cohort, we computed Q_profile on the same contexts for sequences generated by the QMPA model and by the model fine-tuned without the quality-matching constraint. The table below presents the supplementary results.

Q_profile comparison on a common cohort

ArchitectureBaseQMPAnoQ
ESM347.4692 [45.5678, 49.7408]48.4256 [46.0563, 51.3729]47.0077 [44.6992, 49.8306]
PoET-229.4934 [28.1766, 30.8825]40.0968 [37.5367, 42.3906]33.3719 [31.9203, 34.8152]

Paired Q_profile changes (95% confidence intervals)

The paired analysis used 200 common contexts per architecture. We averaged the scored draws within each context and resampled 76 ESM3 and 70 PoET-2 homology components 10,000 times.

ArchitectureQMPA − Base (95% CI)noQ − Base (95% CI)QMPA − noQ (95% CI)
ESM3+0.9564 [0.2455, 1.7116]−0.4615 [−1.0507, 0.1601]+1.4179 [1.1046, 1.7324]
PoET-2+10.6034 [8.2883, 12.8631]+3.8785 [3.1658, 4.5661]+6.7249 [4.7715, 8.6134]

For ESM3, QMPA improved Q_profile by 0.9564 over Base and by 1.4179 over noQ. For PoET-2, QMPA improved Q_profile by 10.6034 over Base and by 6.7249 over noQ. Comparing the Q_profile scores of QMPA and noQ on a common cohort for ESM3 and PoET-2 yields the same conclusion as the original manuscript. QMPA's quality matching contributes to preserving the Q_profile scores of generated sequences.

The supplementary quality-matching ablation resolves the cohort inconsistency in the original manuscript and can serve as the main evidence for the contribution of quality matching in QMPA. We will use this experiment to replace the current Q_profile portion of Section 4.5 in the main text.

AuthorsOpenReview ↗

Response to Reviewer 5F4F — Q3

In the PLM-CARE framework, MVA assesses virulence using VirulentHunter as the preference-labeling classifier, with its role confined to labeling, and DTVF and VP2 as external evaluators, with their role confined to evaluation. The performance of the two external evaluators on natural sequences is therefore something we need to consider. If these evaluators were unreliable, it would be difficult to interpret PLM-CARE's measured effects.

Thank you for pointing out the need to investigate external evaluator reliability, which we had overlooked. To address this question, we audited the default classifications made by DTVF and VP2 on natural proteins using the same frozen configurations as in the generated-sequence evaluation. We matched UniProt-annotated virulence factors to reviewed bacterial enzymes from the same genus and of similar length, forming a time-filtered panel with 88 sequences per group and a larger panel with 288 per group.

Positive-call rates on natural proteins by annotation group

PanelSequences per groupEvaluatorPositive rate for VF-annotated proteinsPositive rate for enzyme controls
Time-filtered88DTVF83.0%42.0%
Time-filtered88VP285.2%62.5%
Larger288DTVF86.1%59.0%
Larger288VP291.7%36.1%

In the two panels, DTVF's positive-call rates for VF-annotated sequences were 83.0% and 86.1%, respectively, and its rates for the enzyme controls were 42.0% and 59.0%, respectively. The corresponding VP2 rates were 85.2% and 91.7% for the VF-annotated group and 62.5% and 36.1% for the controls. Both evaluators were more likely to call VF-annotated natural proteins positive, although positive-call rates among the enzyme controls were substantial and varied across panels. Because the enzyme controls lack experimental confirmation as non-virulent negatives, we report the values as positive-call rates by annotation group. Overall, both external evaluators called a higher proportion of the VF-annotated proteins positive than of the enzyme controls.

The audit of the external evaluators on natural proteins helps interpret what their scores can tell us. We will add the audit to Section 4.2 to explain the external evaluators and to Appendix A.4 to describe where the evaluators come from and the scope of their use.

Reviewer_ZzgXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PLM-CARE, which uses quality-matched preference learning to reduce predicted virulence-factor (VF) association in sequences generated by ESM3 and PoET-2. Fine-tuning lowers predictions from two external classifiers without observed degradation in sequence-quality proxies, although these results do not establish reduced biological risk or preserved function.

Critical strengths
  • Relevant problem and motivated design. The study complements target-recovery evaluation with VF-association assessment, while quality matching addresses the possibility that lower risk scores merely reflect degraded sequence quality.
  • Reasonably strong empirical validation. Evaluation uses fresh generations on held-out contexts and two external classifiers that do not supply training labels, with directionally consistent results across two architectures and three training seeds.
Critical weaknesses
  • The methodological contribution is not fully isolated. The framework primarily integrates existing techniques, and comparisons with simpler baselines, such as supervised fine-tuning on preferred sequences, are needed to establish its added value.
  • Both risk reduction and quality preservation rely on proxies. Classifiers may share data biases, while quality metrics mainly assess family compatibility. These measures do not establish reduced biological risk, preserved function, or retained general design capabilities.
Questions for authors
  • Under matched data and training budgets, does QMPA offer clear advantages over supervised fine-tuning on preferred sequences and preference optimization without quality matching?
  • Why do baseline quality scores differ in the quality-matching ablation? Could the authors use fixed evaluation contexts and report both external risk and quality metrics?
  • Do the improvements generalize to contexts not selected for preference-pair eligibility and to standard benign design tasks? Can the authors provide validation beyond the current classifiers and family-compatibility metrics?
6 comments
AuthorsOpenReview ↗

Response to Reviewer ZzgX - Q1

Thank you for recognizing the strengths of our evaluation and for your constructive suggestions.

On the same held-out contexts, QMPA achieved lower mean DTVF scores and higher mean Q_profile scores than chosen-only supervised fine-tuning (SFT) for both ESM3 and PoET-2. We trained the SFT baseline using only the preferred sequence from each QMPA pair and evaluated both methods on the same held-out contexts. For PoET-2, both methods used 600 optimizer steps. We will describe the experimental configuration in detail in the revised appendix. The table below presents the results.

Comparison of QMPA and chosen-only SFT

ArchitectureMethodMean DTVFΔDTVF vs. Base (95% CI)Mean Q_profileΔQ_profile vs. Base (95% CI where available)
ESM3QMPA0.5166−0.1491 [−0.2510, −0.0731]48.4256+0.9564 [0.2455, 1.7116]
ESM3Chosen-only SFT0.6478−0.0179 [−0.0353, −0.0052]47.1185−0.3507 [−0.5221, −0.2129]
ESM3Base0.6656047.46920
PoET-2QMPA0.7351−0.1405 [−0.1819, −0.0955]40.0968+10.6034 [9.3642, 11.8480]
PoET-2Chosen-only SFT0.9540+0.0784 [0.0673, 0.0900]22.9128−6.5806
PoET-2Base0.8756029.49340

Direct comparison of QMPA and chosen-only SFT

ArchitectureΔDTVFΔQ_profileΔVirulentHunter
ESM3−0.1312 [−0.2173, −0.0657]+1.3071 [0.6155, 2.1325]−0.1894 [−0.2406, −0.1450]
PoET-2−0.2189 [−0.2424, −0.1965]+17.1840 [15.6148, 18.7695]+0.0564

Compared with SFT, QMPA changed DTVF by −0.2189 (95% CI [−0.2424, −0.1965]) and Q_profile by +17.1840 (95% CI [15.6148, 18.7695]) for PoET-2. The corresponding analysis for ESM3 also favored QMPA on DTVF (−0.1312) and Q_profile (+1.3071).

The supplementary chosen-only SFT experiment provides a direct and easy-to-interpret baseline for QMPA. We will report this baseline experiment in Section 4.

We report the supplementary experiment on preference optimization without quality matching in our response to Q2.

AuthorsOpenReview ↗

Response to Reviewer ZzgX — Q2

The baseline quality scores differ from those in the original quality-matching ablation in Section 4.5 because the original ablation used a separately constructed held-out cohort. As a result, Base had different means across cohorts. This creates a problem because subtracting each cohort's own Base score can leave differences between the evaluation cohorts in the comparison. The effect of fine-tuning may also vary by context. We therefore fully agree with your suggestion to fix the evaluation contexts.

We added a quality-matching ablation on a common cohort. In the new analysis, Base, QMPA, and noQ use common contexts and a shared Base reference within each architecture. In a separate homology-restricted analysis, we excluded contexts with homology links to the preference training or validation data of either pipeline. The table below reports the common-cohort results. ESM3 VP2 is reported on the 129 contexts scored in all three arms.

Quality-matching ablation on a common cohort

ArchitectureMethodQ_profileΔQ_profile vs. BaseDTVFVP2 positive rateVH
ESM3Base47.4692 [45.5678, 49.7408]00.666 [0.556, 0.765]74.3% [56.1%, 89.9%]0.604 [0.560, 0.654]
ESM3QMPA48.4256 [46.0563, 51.3729]+0.95640.517 [0.367, 0.666]56.9% [41.7%, 76.7%]0.377 [0.289, 0.457]
ESM3noQ47.0077 [44.6992, 49.8306]−0.46150.614 [0.500, 0.727]62.6% [47.2%, 81.8%]0.466 [0.413, 0.518]
PoET-2Base29.4934 [28.1766, 30.8825]00.876 [0.843, 0.904]82.8% [79.5%, 85.7%]0.553 [0.498, 0.605]
PoET-2QMPA40.0968 [37.5367, 42.3906]+10.60340.735 [0.668, 0.801]61.8% [55.2%, 68.0%]0.408 [0.361, 0.463]
PoET-2noQ33.3719 [31.9203, 34.8152]+3.87850.900 [0.841, 0.942]72.5% [64.4%, 79.7%]0.290 [0.229, 0.369]

Comparing Q_profile for QMPA and noQ on the same cohort for ESM3 and PoET-2 yields a conclusion similar to the one in the original manuscript. Removing the quality-matching constraint lowers the fine-tuned models' generation quality as measured by Q_profile relative to QMPA.

At the same time, the external evaluators showed that the score reductions from fine-tuning were smaller when we removed the quality-matching constraint. We therefore investigated this pattern. We computed VirulentHunter (VH) scores before and after fine-tuning for both methods. VirulentHunter is the classifier used to construct preference pairs during training, so examining its output alongside that of the external evaluators can shed more light on the result. QMPA had the lowest point estimates of the three arms on both external evaluations, DTVF and VirulentPred 2.0 (VP2), for both architectures. This provides evidence that QMPA's improvements transfer better across evaluators. For ESM3, all three measures followed the same trend. The mean scores for DTVF, VP2, and VirulentHunter were lowest for QMPA, followed by noQ and Base. For PoET-2, noQ achieved the largest improvement on VirulentHunter, the classifier used to construct the preferences, but this advantage did not carry over to the external evaluators. In fact, noQ had a higher DTVF risk score than Base. A drop in VirulentHunter alone is therefore insufficient to judge the effect of alignment. QMPA's value lies in retaining an improvement on VirulentHunter while achieving more consistent improvements in the average external scores.

The new quality analysis can replace the current Q_profile portion of Section 4.5. We will also report the external evaluator scores and the preference-labeling classifier scores to extend the conclusions of the ablation.

AuthorsOpenReview ↗

Response to Reviewer ZzgX — Q3

The manuscript and supplementary experiments show that QMPA was effective in the virulence-factor protein design setting we defined. We understand your concern. To show that QMPA preserves the models' generation capabilities, we conducted supplementary experiments on a broader set of contexts and on utility for benign tasks.

To test further how much the results depend on the preference-pair screening procedure, we sampled from the upstream pool without requiring contexts to have entered the preference-pair set. We separately screened for overlap with the training and validation data. The results of this supplementary experiment are presented below.

After screening and excluding samples that overlapped with the training and validation sets, 171 PoET-2 contexts remained. Relative to Base, QMPA changed DTVF by −0.1086 ([−0.1215, −0.0959]) and Q_profile by +2.565 ([1.723, 3.478]). The results show that the joint finding still holds in this broader cohort of natural contexts.

For standard benign design tasks, we used measured GFP fluorescence (1,000 variants) and HXK4 activity (800 variants) to evaluate zero-shot variant ranking. The changes in Spearman's ρ for GFP and HXK4 were −0.0043 and −0.0223 for ESM3, respectively, and +0.1757 and −0.0112 for PoET-2, respectively. The lower bounds of all four 95% confidence intervals exceeded the prespecified threshold of −0.05, so both tasks met the criterion for retention of ranking ability.

Beyond classifier scores and family-compatibility scores, we used AlphaFold 3 to evaluate 344 selected, quality-matched sequence pairs with improved DTVF scores. The mean change in pTM was +0.0013 ([−0.0055, 0.0081]) for ESM3 and +0.0305 ([0.0117, 0.0515]) for PoET-2. The predicted structural confidence of the selected samples provides evidence that the models retained the structural aspect of their generation capabilities.

The three supplementary experiments in Q3 address the question of whether the models retain their generation capabilities more directly. We will incorporate them into the current Section 4.3 as a dedicated part of the manuscript on quality assessment.

Comments on the paper

3 comments

Hi reviewers,

Thank you for your reviewing service. The authors have provided fairly extensive rebuttals. Please confirm that you have read them, state which of your concerns are or are not addressed, and whether that changes your recommendations.

AuthorsOpenReview ↗

Summary of the contribution and rebuttal revisions

Dear Area Chair,  Thank you for coordinating the discussion and for considering our revised submission. Our starting point is a finding about protein language models (PLMs): under standard conditional generation on the evolutionary contexts we studied, both ESM3 and PoET-2 produced sequences that virulence-factor (VF) classifiers flagged as positive. During the rebuttal period, we extended our evaluation to four additional base models—ESM-2 650M, ProtBert, PoET-1, and ProFam-1—and observed classifier-positive sequences for each. Generation used each model's native decoding procedure, with no targeted attack or VF-score guidance. This makes the distribution of outputs under ordinary model use an important target for computational biosafety assessment. The findings concern classifier-predicted VF association in the evaluated contexts.  We introduce PLM-CARE to measure these signals and test whether they can be reduced. The framework combines evolutionary-context prompting for masked and autoregressive PLMs, VF assessment with two external classifiers held out from training, and quality-matched preference alignment (QMPA) adapted to each model's scoring procedure. After alignment, both external classifiers showed average decreases for both models across three training runs. The revisions strengthen the evidence for the roles of quality matching and preference direction, and clarify which capabilities were retained. 

  • Stronger controls and a simpler baseline. We replaced the original cross-cohort ablation with a comparison on shared held-out contexts and added a supervised fine-tuning (SFT) baseline trained on the preferred sequences alone. For both models, QMPA yielded lower mean DTVF scores and higher profile compatibility than either SFT or preference optimization without quality matching. In the PoET-2 random-preference control, QMPA also produced a 0.145 lower mean DTVF score and a 16.29-percentage-point lower VirulentPred 2.0 positive-call rate. (Section 4.4, Tables 2–3 and 19.) 
  • Clearer interpretation of the VF predictions. Both external classifiers showed average decreases, but on shared PoET-2 contexts, directional agreement was only slightly above the rate expected from their marginal decrease rates (77.4% versus 76.0%). The natural-protein audit also found substantial positive-call rates among enzyme controls. Together, these results support a decrease in classifier-predicted VF association under the evaluated conditions. The biological significance of the decreases remains open. (Section 4.2 and Appendices A.4.1 and A.8–A.9.) 
  • Additional evidence on capability retention and evaluation scope. Both models met the prespecified retention criterion for zero-shot variant ranking on GFP and HXK4. Mean predicted structural confidence was maintained across 400 selected sequence pairs, while within-context sequence dissimilarity decreased by about 10% relative to baseline for both models. On 171 additional PoET-2 contexts sampled without requiring inclusion in the preference-pair set, QMPA again lowered mean DTVF and improved profile compatibility. (Sections 4.2–4.3 and Appendix A.7.)  Thank you for considering the revised manuscript and our detailed responses to the reviewers.  Best regards,
    The Authors

Dear Area Chair,

Thank you for your earlier note encouraging the reviewers to read our rebuttal and indicate which concerns remain. We appreciate your help in facilitating the discussion.

With only a few hours left in the discussion period, we would be grateful for any brief feedback the reviewers are able to share. If there are remaining concerns or questions, we would welcome the opportunity to address them before the discussion closes and will do our best to respond in the time available.

Thank you again for your time and for coordinating the review process.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.