acceptodds
Under review as a conference paper at ICLR 2027

Label-Free Fine-Tuning: Your Protein Language Model is Implicitly a Reward Model

Abstract

Post-training of protein language models (PLMs) typically requires verifiable labels from simulations or wet-lab experiments, which are costly to acquire in real world. To overcome the supervision bottleneck, this paper investigates the emerging paradigm of label-free fine-tuning (LFT) to improve protein generation quality and fidelity without ground-truth labels. Prior works have explored various proxy rewards for LFT, yet their design suffer from inconsistent performance across sampling schemes and downstream tasks. To this end, we propose pointwise mutual information (PMI) as a principled reward, which measures the difference between the conditional and unconditional log probability of a given PLM. Despite its simplicity, PMI is motivated by Bayes' rule to effectively leverage a well-trained PLM as an implicit reward model, analogous to classifier-free guidance in diffusion models. Extensive experiments also demonstrate that our framework is amenable and transferable to a broad class of PLMs and protein design tasks.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Protein Language Models, Generative Models, Post-Training, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xi2YOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates pointwise mutual information (PMI) as a reward for label-free fine-tuning of protein language models (PLMs). The authors' main proposal is that the difference between conditional and unconditional sequence log-likelihoods can measure compatibility with a design prompt, allowing a pretrained PLM to act as an implicit reward model without external supervision. Motivated by Bayes' rule and connections to classifier-free guidance, PMI is proposed as a more transferable measure of prompt fidelity across protein design tasks. The training procedure generates candidate sequences from a pretrained PLM given a design prompt, ranks them by PMI (how much more likely conditioned on prompt vs unconditional), and assigns positive and negative labels to the top and bottom candidates, respectively. These pseudo-labels are used for binarized reward optimization (BRO), which adjusts the log-likelihood ratio between the fine-tuned and reference models to favor positive over negative samples. Experiments across ProGen2 and ESM3 report correlations with experimentally measured protein properties, more consistent reward discrimination across sampling temperatures than raw likelihood scores, and improved conditional generation fidelity after fine-tuning, and generally outperforming the L1 proxy reward across the reported evaluation metrics.

Critical strengths

Simple and broadly applicable reward formulation The proposed PMI reward provides a mathematically motivated way to estimate prompt compatibility directly from a pretrained PLM, without requiring task-specific labels or a separate reward model. Its simplicity and applicability across different conditioning modalities make it a potentially useful approach to label-free post-training.

Encouraging empirical results across multiple settings The experiments span different PLM architectures, functional and structural conditioning tasks, and sampling temperatures. PMI exhibits relatively consistent reward discrimination across temperatures and improves generation fidelity over the base models, generally performing competitively with or better than the L1 baseline. The retrospective correlations with experimentally measured protein properties provide additional evidence that PMI captures biologically relevant information.

Critical weaknesses

Positioning and contributions relative to prior work. The paper would benefit from a more substantive discussion of its relationship to two closely related works that it already cites: ProteinZero (Wang et al., 2025) and Be Your Own Teacher (Li et al., 2026). ProteinZero already explores conditional/unconditional likelihood contrasts as part of a more elaborate reward formulation for protein design, while Be Your Own Teacher establishes the use of BRO with alternative proxy rewards for label-free fine-tuning. The present paper offers a considerably cleaner formulation, which is a strength. However, given these precedents, it is important to clarify what new insights or capabilities the PMI formulation provides beyond demonstrating that it is another effective proxy reward. In particular, I would like to better understand which aspects of the proposed approach are substantively new and which are simplifications or extensions of existing ideas.

Insufficient analysis of why PMI-based fine-tuning works. Given the connections to prior work, the clean formulation presents an opportunity for more targeted ablations and analyses to establish the mechanism behind the improvements. Most notably, the fine-tuning experiments do not compare PMI+BRO against conditional-likelihood+BRO. There are also broader questions about the self-training procedure, such as amplifying existing model biases or narrowing the generation distribution and reducing diversity. These concerns are motivated by observations in prior work: ProteinZero investigates diversity collapse in protein-model reinforcement learning, while Be Your Own Teacher reports diminishing improvements at larger sampling budgets, suggesting that BRO may primarily increase the probability of generating already-accessible successful outputs. More broadly, it begs the question of what capabilities must already be latent in the pre-trained PLM for this approach to work.

Questions for authors

The two weaknesses above are my questions. I am open to increasing my score, pending the authors' response.

Reviewer_uxUZOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose using pointwise mutual information (PMI) as a reward to fine-tune protein language models without additional experimental labels. The reward compares the likelihood of a sequence with and without a conditioning prompt, using the same pretrained model for generation and scoring. Sequences with high and low rewards are then used as positive and negative examples for fine-tuning with the existing BRO algorithm. This is similar in nature to the original DPO. They evaluate the score for predicting mutational effects on antibody binding and protein stability, and test conditional generation with ProGen2 and ESM3. The authors report improvements in how well the generated sequences follow the requested functional and structural conditions.

Critical strengths

The authors improve conditional generation using the pretrained model itself to score its outputs, without new experimental labels or a separate reward model. For ESM3, secondary-structure agreement increases from 0.719 to 0.799 and SASA correlation from 0.746 to 0.808.

Critical weaknesses

I am not fully convinced by the conceptual contribution. The framing reads very similarly to Rafailov’s DPO, although the likelihood ratios have different meanings. BRO is also taken from previous work. It should be clearer whether the contribution is primarily the reward, and what this adds beyond existing conditional–unconditional guidance.

The evaluation does not sufficiently establish the claimed generality. The paper should establish benchmarks against previous methods. For example, the mutational-effect experiments lack a clear comparison with ProteinGym, another that comes to mind is Pugh et al.’s likelihood-fitness bridging approach (NeurIPS 2025). The generation results are mixed relative to competing methods.

Very seriously: Table 2 currently says the following: "Novelty values other than PMI are synthetic placeholders; their rankings are illustrative." Are the authors inserting synthetic data in the benchmarks? This is not an appropriate submission.

Questions for authors

Correction of reported results: What are the actual novelty measurements replacing the placeholders in Tables 2 and 3, and do they support the stated conclusions?

Additional evaluation relevant to the current claims: How does PMI compare with standard likelihood scoring methods?

What is the major contribution of the manuscript? The authors need to make an effort to write it in a manner that is approachable to the audience. It reads as if considerable AI was involved.

Reviewer_diAeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes pointwise mutual information (PMI), the gap between a protein language model's conditional and null-conditioned sequence log-probabilities, as a label-free reward for post-training. Candidates are sampled from a reference model, labeled by frozen PMI, and fine-tuned with binarized reward optimization (BRO). The authors interpret PMI as an implicit prompt-compatibility score related to mutual information and classifier-free guidance. Experiments cover zero-shot mutational-effect prediction, reward diagnostics across sampling temperatures, and fine-tuning ProGen2 and ESM3 for function- and structure-conditioned generation.

Critical strengths

The proposed reward is simple, which uses one frozen protein language model and requires no external encoder or experimental labels. The reward diagnostics also give a concrete reason to prefer the conditional–null gap over raw log-probabilities, whose association with keyword recovery changes across sampling temperatures.

Critical weaknesses
  1. Serious concerns regarding synthetic results in Table 2 and 3.

I believe this paper warrants a desk rejection due to the inclusion of synthetic results presented alongside actual experimental results in Tables 2 and 3. The caption of Table 2 explicitly states:

"Novelty values other than PMI are synthetic placeholders; their rankings are illustrative."

Similarly, the caption of Table 3 states:

"Novelty values other than Base and PMI are synthetic placeholders; their rankings are illustrative."

Despite these explicit acknowledgments, the synthetic values are still included in the main experimental comparisons alongside genuine results, with some even highlighted using boldface and underlining. This raises serious concerns that parts of the reported results may have been generated by AI rather than obtained through actual experiments.

  1. The derivation in Section 3.2 does not describe the algorithm that is actually trained.

Equation (11) identifies the conditional–unconditional gap with the mutual information Iπθ(X;Y)I_{\pi_\theta}(X;Y). That identity requires the marginal π(y)=∑xp(x)π(y∣x)\pi(y)=\sum_x p(x)\pi(y\mid x) in the denominator. The implementation instead subtracts a null-conditioned score π(y∣∅)\pi(y\mid\emptyset). Appendix A.4 correctly describes this score as an operational baseline, so the first term of Equation (11) is not the mutual information. Replacing the marginal with π(y∣∅)\pi(y\mid\emptyset) also removes the exact Bayes identity used to interpret the score as an implicit classifier p(x∣y)p(x\mid y).

Equation (4) gives the optimal policy only when the reward is external to the optimized policy. In Equations (11)–(12), the reward depends on πθ\pi_\theta, while the right-hand side still contains that same πθ\pi_\theta. This does not yield the stated closed form. The implemented reward is different again, which is computed once with the frozen reference model.

  1. Inconsistency between the claims and results in Table 1.

The authors claim that PMI “achieves competitive keyword recovery and consistently better diversity, novelty and pLDDT/pTM” than L1. Table 1 does not support the claim of consistent improvement. L1 achieves higher pLDDT on ProGen2-small (0.594 vs. 0.582) and higher pTM on ProGen2-medium (0.491 vs. 0.477).

Questions for authors
  1. See Weakness. The author should explain why illustrative values were boldfaced or underlined as best or second-best results in Table 2 and 3.
  2. The paper adopts binarized reward optimization (BRO) from Li et al. (2026) but does not compare it with soft reward optimization (SRO), the other algorithm proposed in that work. Please add SRO to Table 1 under both the L1 and PMI rewards, using the same ProGen2 prompts, temperatures, and evaluation metrics. In Li et al., SRO is already close to BRO on this benchmark, so the comparison is needed to show whether the reported gains come from PMI or from the choice of BRO.

Li, Lanqing, et al. "Be Your Own Teacher: Steering Protein Language Models via Unsupervised Reward Optimization." arXiv preprint arXiv:2606.18961 (2026).

Comments on the paper

1 comment

Tables 2 and 3: "Novelty values other than PMI are synthetic placeholders; their rankings are illustrative."

#4 of https://researchonresearch.blog/2026/10/20/self-confessed-fabrications-and-rise-of-tbds/

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.