acceptodds
Under review as a conference paper at ICLR 2027

Be Your Own Teacher: Steering Protein Language Models via Unsupervised Reward Optimization

Abstract

Protein language models (PLMs) have emerged as powerful tools for controllable biomolecular design, yet their post-training adaptation typically relies on costly wet-lab validation or curated preference datasets. To overcome this supervision bottleneck, we introduce unsupervised reward optimization of PLMs, a comprehensive framework for steerable protein generation without ground-truth labels. Our key insight is that task-agnostic rewards, which combine intrinsic model uncertainty with extrinsic semantic consistency informed by protein representation models, exhibit strong correlation with controllability measures across base models and temperature regimes. Building upon this discovery, we propose two offline algorithms: Soft Reward Optimization (SRO) and Binarized Reward Optimization (BRO), which effectively maximize the classical RLHF objective induced by these proxy rewards. Extensive experiments on compositional out-of-distribution prompts demonstrate that both methods significantly outperform competitive baselines (DPO, KTO), while approaching oracle performance across multiple sampling temperatures, model scales and protein families. Moreover, PLMs fine-tuned with unsupervised rewards can achieve consistently higher coverage compared to their base model in pass@k evaluations. By enabling self-improvement of PLMs through their own generated experience, our framework provides a scalable pathway toward controllable biomolecular design in settings where labeled preferences or experimental feedback are scarce or unavailable.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Protein Language Models, Generative Models, Post-Training, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SDDqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper mainly discusses using model-generated proxy rewards to perform offline post-training (e.g. offline RL) on protein large language models, thereby improving conditional generation performance without requiring task-level ground-truth reward labels for generated samples (which is too slow). The authors propose offline algorithms such as SRO and BRO and conduct experiments across various tasks. The results demonstrate the effectiveness of the proposed methods, with additional results provided in the appendix.

Critical strengths

I find the research on offline RL for PLMs to be quite significant; the authors analyzed the relationship between surrogate rewards and temperature and observed some interesting results/findings, which I consider great This work offers valuable insights into reward selection/design and data sampling.

In general, the proposed method demonstrates effectiveness across few models, and the authors present ablation results for the various components.

Critical weaknesses

Although the paper presents some experiments, there remain significant concerns. The current evaluation seems insufficient to demonstrate the broad applicability of the proposed method for the offline post-training of PLMs, as the tasks addressed appear somewhat limited. Furthermore, the paper fails to show whether the method achieves reliable generalization across a wide range of unseen tasks. I encourage the authors to address these problems in their discussion.

Additionally, I am quiet curious whether the proxy reward relies on final labels (e.g., ground-truth, real label), or if the proposed method can be applied to tasks where ground-truth labels are unavailable. I suggest the authors provide a detailed discussion of the method and experimental setup.

Finally, given the use of a reward proxy, why did the authors not compare their approach against mainstream online RL algorithms, such as GRPO?

Questions for authors

See Critical Weaknesses and,

I recommend that the authors address issues regarding the applicability of the proposed method, its reliance on ground-truth (GT) labels, and the lack of sufficient baselines for offline and online RL.

1 comment

We sincerely thank the reviewer for the constructive review and for identifying some of the key strengths of our paper. For your concerns, we respond point by point. Please also refer to the seperate General Response thread on top for additional experiments and discussion of commonly raised questions.

W1: Applicability of the proposed method

Since reviewer SDDq didn't specify the limitation, we can only audit on the scope of applicability that has been covered in our experiments:

ResultsApplicability Scope
Figure 1& Table 4-9> 20 reward metrics, Func2Seq and Struct2Seq tasks, 2 embedding models
Table 1Autoregressive ProGen2, 2 model scales, compositional OOD prompts, sampling temperature variation, proxy reward and ground-truth labels
Table 107 Pfam families
Table 11hyperparameter variation
Table 15-16Multimodal ESM3, Struct2Seq task

Additionally, in General Response on top, we have demonstrated positive transfer on heldout family and applicability with base-model-generated embeddings. Overall, we think these results have demonstrated broad applicability of our proposed method.

W2: Reliance on ground-truth (GT) labels

We appreciate the concern. However, we must clarify that our benchmarking results in section 4.2 are merely diagnostics to showcase the correlation between proxy rewards and the actual ground truth labels, i.e., to evaluate if these metrics qualify as a proxy of the ground truth labels. This procedure is not required and can be skipped entirely for post-training purpose. As shown in Table 2, simply using the L1 metric across all temperatures achieves 0.433 (BRO) and 0.389 (SRO) in keyword recovery, significantly enhancing the base performance (0.281) and outperforming DPO (0.350) and KTO (0.337) under the same setting. We also provide new results on multi-seed runs (see General Response) to show that the gains are statistically significant. Given that L1 metric is task- and temperature-agnostic by requiring only a representation model to generate sequence embeddings, our proposed post-training method can achieve significant and meaningful gains without any information about the actual ground truth. Table 2 also shows that other simple metrics such as model likelyhood also bring notable gains (BRO 0.341 and SRO 0.364 vs. Base 0.281) without ground truth information, so L1 is not just a lucky or cherry-pick case. Therefore, metrics like L1 and likelihood can be used and effectively improve the base model in a zero-shot, offline setting.

So the key is that post-training procedure is independent of the reward diagnostics, despite the fact that we present it as section 4.3, right after the reward benchmarking as section 4.2. We will adjust our writing in the final version accordingly to make it clear.

W3: Lack of sufficient baselines for offline and online RL

For offline RL, we have included DPO and KTO, which are competitive baselines widely applied for LLMs and protein lanaguages models. These methods are most relevant because they use binary labels and do not require continuous, explicit rewards, just like BRO. We restrict our method and evaluation to offline setting because 1. It is more practical: in protein engineering and biology, there are many unlabeled datasets that could be used, while the labeling is the bottleneck. Also given that a typical protein experiment takes days or even weeks, it's impossible to train a PLM by interactively receiving online feedback from the experiments. 2. For online training with our proposed unsupervised rewards, which can be evaluated much faster, there is the risk of overoptimization: e.g., He et al [1] observed consistent reward hacking and model collapse at later stages when using intrinsic rewards such as model entropies and likelihood. Therefore, in this paper we prioritize offline RL setting and leave online development for future work.

[1] He, Bingxiang, et al. "How Far Can Unsupervised RLVR Scale LLM Training?." International Conference on Learning Representations. Vol. 2026. 2026.

Hope our reply addresses your concerns and clarifies any misunderstanding. If you find them helpful, please consider raising your score, thank you!

Reviewer_2sdZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces two methods, SRO and BRO, that are intended to perform reward-guided optimization for protein language models without any notion of downstream ground truth labels. The authors provide an extensive theoretical discussion that motivates and explains the method. To empirically evaluate its performance, they perform several PLM experiments on ProGen2 and ESM3 models across inverse folding and sequence-level optimization tasks.

Critical strengths

The authors provide extensive background and motivation for their work, and I found everything up until the experimental setup to be quite clear. I also found their choice of two different models and tasks to be a sensible framework for empirical evaluation. Overall, the idea presented in this paper is certainly one I find to be compelling.

Critical weaknesses

The main concern I have with this manuscript is that the results do not fully support the methodological claims that are made in the framing of this work. To my understanding, it does not seem that the rewards are completely unsupervised by design, as the ground-truth labels are needed to determine whether the NLL or embedding distance rewards are most correlated. Furthermore, the viability of this method relies upon the assumption that the pre-trained distribution of a model, whether it be through the NLL or embeddings, is well-aligned with the desired functional optimization. Therein lies what I view as the fundamental limitation of the results, as several of the choices made in the evaluations of the method were either confusing (more on this in the questions section) or just not really directly probing the question of whether such unsupervised rewards can be useful with real, offline experimental protein fitness data.

Questions for authors

Clarification:

  • If determining the optimal reward for a given temperature is done with ground-truth labels, how can this be used in a zero-shot, offline setting?
  • How would you propose navigating settings where fitness and model likelihoods are misaligned?
  • Why do you use ESM-C for the embeddings in the distance-based reward rather than the embeddings from the model you are actually using to generate sequences? I'm not sure I fully understood that choice.
  • Can you elaborate on why you would construct OOD prompts by matching PFAM IDs to proteins from other families? I'm struggling to see how that is analogous to any real relevant use case.
  • What are the error bars on Table 1? I have no gauge on whether the highlighted differences shown are statistically significant.

Suggestions:

  • On a similar note to my last clarification point, what do the results look like in Table 3 if you run them for multiple seeds? Do your claims of significant improvement hold up?
1 comment

We sincerely thank the reviewer for the constructive review and for identifying some of the key strengths of our paper. For your concerns, we respond point by point. Please also refer to the seperate General Response thread on top for additional experiments and discussion of commonly raised questions.

W1&Q1: Reward selection requires ground-truth labels

We appreciate the concern. However, we must clarify that our benchmarking results in section 4.2 are merely diagnostics to showcase the correlation between proxy rewards and the actual ground truth labels, i.e., to evaluate if these metrics qualify as a proxy of the ground truth labels. This procedure is not required and can be skipped entirely for post-training purpose. As shown in Table 2, simply using the L1 metric across all temperatures achieves 0.433 (BRO) and 0.389 (SRO) in keyword recovery, significantly enhancing the base performance (0.281) and outperforming DPO (0.350) and KTO (0.337) under the same setting. We also provide new results on multi-seed runs (see General Response) to show that the gains are statistically significant. Given that L1 metric is task- and temperature-agnostic by requiring only a representation model to generate sequence embeddings, our proposed post-training method can achieve significant and meaningful gains without any information about the actual ground truth. Table 2 also shows that other simple metrics such as model likelyhood also bring notable gains (BRO 0.341 and SRO 0.364 vs. Base 0.281) without ground truth information, so L1 is not just a lucky or cherry-pick case. Therefore, metrics like L1 and likelihood can be used and effectively improve the base model in a zero-shot, offline setting.

So the key is that post-training procedure is independent of the reward diagnostics, despite the fact that we present it as section 4.3, right after the reward benchmarking as section 4.2. We will adjust our writing in the final version accordingly to make it clear.

W2&Q2: Method validity when fitness and model likelihoods are misaligned

The question seems to imply that the proxy rewards are effective only if they are aligned with the ground truth labels/fitness. However, our experiments provide a counterexample: Figure 1 shows that L1 rewards only aligns well with the ground truth at T=0.7, yet Table 1 and 2 show training BRO/SRO with L1 reward sampled across all temperatures can still signficantly improve the model performance (BRO 0.433 (Table 1) and SRO 0.389 (Table 2) vs. Base 0.281), and is more effective than L1 sampled at the best-aligned temperature (Table 3: BRO-multiT 0.433 vs. BRO-T0.7 0.366). Intuitively, our proposed rewards provide confidence-based and self-consistency-based objectives to regularize the base model. One of the key contribution of our empirical study is to show that, even it's not guaranteed and in fact impossible to accurately approximate the ground truth with any heuristic metrics, they can still be used as label-free rewards to effectively improve the controllability of protein language models.

Q3: Use of ESM-C instead of the base model for sequence embeddings

As requested, we provide a new run for BRO trained with L1 metrics computed from the internal embeddings of the same base model, circumventing the use of external models such as ESMC for data generation:

MethodT = 0.3T = 0.5T = 0.7T = 1.0T = 1.5Average
BRO-L1-self34.138.442.543.024.036.4

The results outperform Base/DPO/KTO and underperform BRO-L1 with ESMC embeddings. This suggests that an intrinsic L1 reward is also effective, but better representation model (as opposed to the GPT-style progen2, ESMC was optimized for representation learning) would be more beneficial for post-training.

Q4: Rationale of OOD prompts construction

As explained in the footnote of page 7, matching PFAM IDs to proteins from other families reflects an use case of reprogramming functional context from one protein family onto the N-terminal scaffold of another. In more plain words, one may use such prompts to design a compositional functional protein that includes both a motif of family A while also possessing the funcition of family B. In our case, we are constrained by the autoregressive progen2 to combined all the design objectives as the starting sequence. For other PLMs such as ESM3, one may stack different function/structure objectives via different tracks of the model input, which allows for degree of freedom for both prompt design and model generation.

Q5: Error bars and uncertainties:

Please refer to our General Response on top for multi-seed runs and family-heldout transfer experiments, which demonstrate statistically significant superiority of BRO/SRO compared to baselines like DPO and KTO.

Reviewer_4wq1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors investigate the usage of unsupervised rewards while fine-tuning protein language models for controllable generation. Their main contributions involve the design of task-agnostic intrinsic and extrinsic rewards, incorporating a multi-temperature strategy when creating offline datasets, and two algorithms named SRO and BRO. Their results show that their algorithm outperforms their proposed baselines, but still achieves lower performance compared with supervised baselines.

Critical strengths
  1. The task-agnostic reward function composing uncertainty and self-consistency metrics is interesting as a fine-tuning objective for pLMs.
  2. The multi-temperature sampling strategy and analysis are important from a search perspective.
  3. The authors propose two algorithms SRO and BRO that may be important for the field.
  4. The creation of OOD prompt set is an important addition for evaluation.
Critical weaknesses
  1. The benchmarking for the pLMs seems to use rewards from the fine-tuning stage, which raises questions regarding the robustness of results.
  2. The Struct2Seq results seem to be excluded and not properly investigated in the manuscript.
  3. The limitations on the performance being lower than other proxies like pLDDT and pTM lead to questions on when this method should be applied.
Questions for authors

Comments:

  1. The results obtained show that the performance is lower than that of other proxies from protein structure prediction models. Can the authors discuss further whether the proposed method is to be used as a fine-tuning stage during the training of pLMs, and whether they see the method as a part of design campaigns or if supervised fine-tuning is still needed?
  2. In this direction, it seems that the paper would improve by adding 2-stage where the proposed method is performed before a final supervised fine-tuning stage for a specific design campaign.
  3. The lines 413-416 are a big weaknesses of the manuscript, that show that the method is still dependent on the sampling leading to a good search space. What are the requirements of positive samples for self-improvement?
  4. Also, it is unclear to the reviewer how the Struct2Seq is used. I was expecting to see also comparisons against other structure-to-sequence design methods as an important part to validate the proposed method.
  5. (lines 365-370) The text in these lines seems to show what is expected from this autoregressive stochastic sampling. Additional discussion on these patterns and also a comparison with other sampling strategies and non-regressive alternatives would improve the manuscript.
  6. Can you explain further the pass@k metrics for Table 1 and Fig. 2? Do they use the same proxies from training? The reviewer is concerned that additional orthogonal metrics are needed when comparing these results.
  7. Is there a specific reason that GRPO is not used as a method for comparison? Am I missing any assumption that does not hold for applying GRPO?
  8. Appendix E shows all the distance functions explored, which seem very extensive and crucial for the performance obtained. Additional discussion on how dependent the method is on this search is needed. Does this limit the applicability of the proposed method?

Minor Comments:

  1. The acronyms PLMs and RLHF are defined twice.
2 comments
AuthorsOpenReview ↗

Rebuttal 1

We sincerely thank the reviewer for the constructive review and for identifying some of the key strengths of our paper. For your concerns, we respond point by point. Please also refer to the seperate General Response thread on top for additional experiments and discussion of commonly raised questions.

W1: The benchmarking for the pLMs seems to use rewards from the fine-tuning stage

We take this as a concern for the reliance of reward selection on ground-truth (GT) labels. We must clarify that our benchmarking results in section 4.2 are merely diagnostics to showcase the correlation between proxy rewards and the actual ground truth labels, i.e., to evaluate if these metrics qualify as a proxy of the ground truth labels. This procedure is not required and can be skipped entirely for post-training purpose. As shown in Table 2, simply using the L1 metric across all temperatures achieves 0.433 (BRO) and 0.389 (SRO) in keyword recovery, significantly enhancing the base performance (0.281) and outperforming DPO (0.350) and KTO (0.337) under the same setting. We also provide new results on multi-seed runs (see General Response) to show that the gains are statistically significant. Given that L1 metric is task- and temperature-agnostic by requiring only a representation model to generate sequence embeddings, our proposed post-training method can achieve significant and meaningful gains without any information about the actual ground truth. Table 2 also shows that other simple metrics such as model likelyhood also bring notable gains (BRO 0.341 and SRO 0.364 vs. Base 0.281) without ground truth information, so L1 is not just a lucky or cherry-pick case. Therefore, metrics like L1 and likelihood can be used and effectively improve the base model in a zero-shot, offline setting.

So the key is that post-training procedure is independent of the reward diagnostics, despite the fact that we present it as section 4.3, right after the reward benchmarking as section 4.2. We will adjust our writing in the final version accordingly to make it clear.

W2&3: Struct2Seq performance

We did perform Struct2Seq evaluation in Table 15 and 16, where BRO outperforms the base, inference-time scaling (best-of-n) and getting close to supervised baseline. We didn't discuss it extensively because according to reward diagnostics in Figure 1, pLDDT and pTM seems to be strictly better than our proposed heuristic metrics on Struct2Seq tasks, even though they are much more costly to evaluate (on a single A40 GPU, A40 GPU, the average inference time per sequence for ESMC-600M is less than 0.1s, whereas ESMfold takes more than 2.0s). However, this doesn't mean our proposed method is not useful for Struct2Seq. As requested, we provide new Struct2Seq evaluations for ESM3-1.4B as follows (only 1 seed was run due to time constraint)

3D Coordinates conditional generation:

ModelMethodKeyword Recovery ↑
ESM3 (1.4B)Base0.197
ESM3 (1.4B)BRO-L10.255
ESM3 (1.4B)BRO-L1 Best-of-640.224
ESM3 (1.4B)Supervised0.260

Secondary structure (SS8) conditional generation:

MethodAccuracy ↑
Base0.719
BRO-L10.735
BRO-L1 Best-of-640.728
Supervised0.594

Solvent accessible surface area (SASA)conditional generation:

MethodSpearman ↑
Base0.746
BRO-L10.754
BRO-L1 Best-of-640.752
Supervised0.807

which demonstrates the effectiveness of our proposed method BRO-L1.

Q1Q2 Good questions. One key insight of the paper is that, although it's not guaranteed and in fact impossible to accurately approximate ground truth, we have identified various statistical metrics that can be used as label-free rewards to improve the controllability of protein language models. Intuitively, our proposed rewards provide confidence-based and self-consistency-based objectives to regularize the model. Since our experiments have demonstrated broad applicability of the proposed methods (see discussion in our rebuttal to Reviewer SDDq) including Struct2Seq tasks, we expect the methods to serve as a general label-free fine-tuning procedure for PLMs in scenarios where no labels/very few labels are accessible. However, as shown in Table 1 and Figure 2, our methods still lag behind the supervised baseline, so it cannot replace supervised fine-tuning. Your 2-stage suggestion sounds very promising and we will include it in the final version should the paper be accepted. Thank you for the suggestion!

Q3 This presents a smilar issue for self-improvement of general LLMs. We think the general consensus is that, an ideal candidate for self-training/distillation would be a medium-level model. It should be good enough to generate plausible samples for labeling and training, while it doesn't need to be the best-performing, which leaves little room for improvement. Our choices of ProGen-151M/764M and ESM3-1.4B reflect exactly the rationale.

AuthorsOpenReview ↗

Rebuttal 2

Q4 As requested, we have provided Struct2Seq comparison between BRO-L1 and inference-time scaling (best-of-n) and supervised baselines. Since our proposed method is for post-training, we need to compare to other structure-to-sequence design methods that are compatible on the same base model. Due to rebuttal time constraint, we will explore more baselines in the final version should the paper be accepted. Thank you for the suggestion!

Q5 Please check Table 3 for comparison with single-temperature sampling strategies, and Table 15, 16 and new Struct2Seq experiments above for non-autoregressive ESM3 results.

Q6 Table 1 and Fig. 2 use exactly the same proxies, but examine the performance from different angles. Table 1 can be interpreted simply as "pass@1" in Figure 2, which measures the average success rate of generated sequence given a prompt. pass@k measures the probability that at least one of k generated candidates passes a predefined success criterion. It evaluates whether a model can produce a successful answer within k attempts. pass@1 reflects single-attempt reliability; larger k measures success with additional attempts. Please refer to [1] for formal motivation and mathematical definition.

[1] Chen, Mark, et al. "Evaluating large language models trained on code." arXiv preprint arXiv:2107.03374 (2021)

Q7 GRPO is an online RL method an offline adaptation is needed to properly implement it on our offline setting, which is out of the scope of this paper. We restrict our method and evaluation to offline setting because 1. It is more practical: in protein engineering, there are many unlabeled datasets that could be used, while the labeling is the bottleneck. Also given that a typical protein experiment takes days or even weeks, it's impossible to train a PLM by interactively receiving online feedback from the experiments. 2. For online training with our proposed unsupervised rewards, which can be evaluated much faster, there is the risk of overoptimization: e.g., He et al [2] observed consistent reward hacking and model collapse at later stages when using intrinsic rewards such as model entropies and likelihood. Therefore, in this paper we prioritize offline RL setting and leave online development for future work.

[2] He, Bingxiang, et al. "How Far Can Unsupervised RLVR Scale LLM Training?." International Conference on Learning Representations. Vol. 2026. 2026.

Q8 Appendix E provides additional diagnostics of over 20 metrics. This does not limit the applicability of the proposed method, in contrast, it show there exists a broad spectrum of statistical metrics as candidate proxy rewards for label-free fine-tuning. Through Table 1-3, we have shown that L1-mean, model likelihood (entropy) can effectively improve model controllability across diverse settings, and we encourage future works to explore the other metrics as well as those not covered in the paper.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Reponse to Common Concerns 1

We sincerely thank the reviewers for their time and constructive review. For commonly raised concerns and questions that are likely to influence the decision, we provide a general response as follows.

  1. Ground-truth dependence. Reviewers are worried that the reward selection may require access to the ground truth label, as shown in Figure 1 and Table 4-9. However, we must clarify that these results are merely diagnostics to showcase the correlation between proxy rewards and the actual ground truth labels, i.e., to evaluate if these metrics qualify as real proxy of the ground truth labels. We want to emphasize that this procedure is not required and can be skipped entirely for post-training purpose. As shown in Table 2, simply using the L1 metric across all temperatures achieves 0.433 (BRO) and 0.389 (SRO) in keyword recovery, significantly enhancing the base performance (0.281) and outperforming DPO (0.350) and KTO (0.337) under the same setting. We also provide new results on multi-seed runs (see next section) to show that the gains are statistically significant. Given that L1 metric is task- and temperature-agnostic by requiring only a representation model (as reviewer 2sdZ pointed out, one can simply use the base model as well, for which we provide new positive evidence in the following section) to generate sequence embeddings, our proposed post-training method can achieve significant and meaningful gains without any information about the actual ground truth.

Remark Table 2 also shows that other simple metrics such as model likelyhood also bring notable gains (BRO 0.341 and SRO 0.364 vs. Base 0.281) without ground truth information, so L1 is not just a lucky or cherry-pick case. The piecewise reward rTr_T in eqn (28) does require ground truth information to calibrate, which brings more gain for SRO but underperforms the simple L1 reward for BRO. We include this design to suggest that, given some ground truth information, one might be able to better calibrate the reward and further improve the post-training result in some cases. However, by our results on L1 and likelihood, the proposed method can work well in an entirely label-free fashion.

Reviewer 2sdZ questioned our method's validity when fitness and model likelihoods are misaligned, which warrants a clarification. One key contribution of our empirical study is to show that, even it's not guaranteed and in fact impossible to accurately approximate ground truth with any metrics, they can still be used as label-free rewards to improve the controllability of protein language models. Experimental evidence: Figure 1 shows L1 rewards only aligns well with the ground truth at T=0.7, yet Table 1&2 show training BRO/SRO with L1 reward across all temperatures can signficantly improve the model performance (BRO 0.433 (Table 1) and SRO 0.389 (Table 2) vs. Base 0.281), and is more effective than L1 sampled at the best-aligned temperature (Table 3: BRO-multiT 0.433 vs. BRO-T0.7 0.366). Intuitively, our proposed rewards provide confidence-based and self-consistency-based objectives to regularize the base model.

  1. Additional repeated-run evidence. To better gauge the statistical significance of our experiments, we provide the following supplementary summary from the additional Progen2-small-mix7 runs on the same compositional OOD prompts. The results are in percentage and evaluated with 3 seeds. Each run independently includes training, generation, and evaluation.
MethodT = 0.3T = 0.5T = 0.7T = 1.0T = 1.5Average
Base21.1 ± 0.128.7 ± 0.839.5 ± 0.140.0 ± 0.312.1 ± 0.428.3 ± 0.1
DPO-L125.9 ± 0.735.2 ± 1.047.7 ± 2.044.9 ± 0.513.1 ± 0.133.4 ± 0.7
KTO-L126.7 ± 1.034.2 ± 0.643.7 ± 0.346.6 ± 0.616.1 ± 0.333.5 ± 0.5
BRO-L140.7 ± 0.143.9 ± 0.849.1 ± 0.851.3 ± 0.230.3 ± 0.243.0 ± 0.3
SRO-L134.7 ± 0.139.2 ± 0.146.3 ± 0.551.5 ± 0.127.8 ± 0.339.9 ± 0.1
DPO-GT30.4 ± 0.543.9 ± 0.959.1 ± 0.455.9 ± 1.017.6 ± 0.441.4 ± 0.7
KTO-GT29.2 ± 0.345.0 ± 0.962.1 ± 0.163.6 ± 0.126.9 ± 0.445.3 ± 0.3
BRO-GT47.8 ± 0.150.9 ± 0.152.6 ± 0.350.1 ± 0.128.1 ± 0.345.9 ± 0.1

Entries like DPO-L1 and DPO-GT stands for base model trained via DPO on L1 and ground-truth labeled datasets to seperate the effect of data and optimization algorithm. SRO-L1 was intentionally retrained using fixed L1 rewards intead of the piecewise rTr_T reported in Table 1 of the paper for label-free evaluation. On both datasets, DPO/SRO significantly outperforms Base/DPO/KTO, suggesting an algorithm advantage. By comparing the same algorithm on both datasets, we see that ground truth labels do bring more gains after post-training, and BRO is particularly robust to noisy reward evidenced by a small gap (43.0 vs 45.9) between L1 and GT datasets.

AuthorsOpenReview ↗

General Reponse to Common Concerns 2

  1. Post-training transfer and scope. To demonstrate broader applicability and generalization, we perform an additional study with a more challenging data split: training the model on 6 Pfam families (600 prompts), leaving PF00042 (100 prompts) as a heldout-test sets. This setting tests the generalization of our method on unseen families (the PF00042 was seen during SFT, but the OOD prompt set consisting of PF00042 family tag + first 10 tokens of PF00125 seqs is unseen during either SFT or post-training).
MethodT = 0.3T = 0.5T = 0.7T = 1.0T = 1.5Average
Base11.4 ± 0.415.8 ± 0.123.6 ± 0.725.9 ± 0.98.31 ± 0.717.0 ± 0.2
DPO14.4 ± 1.020.9 ± 1.034.1 ± 1.031.7 ± 0.88.57 ± 0.121.9 ± 1.0
KTO15.2 ± 2.020.9 ± 0.329.1 ± 0.431.2 ± 2.011.6 ± 0.821.6 ± 0.1
BRO23.1 ± 1.026.5 ± 0.531.4 ± 0.133.4 ± 1.019.1 ± 0.926.7 ± 0.3
SRO16.5 ± 0.421.4 ± 0.328.3 ± 1.031.1 ± 0.117.4 ± 0.523.0 ± 0.3

The ranking is similar to the normal setting: BRO > SRO > DPO/KTO > Base. Additionally, as suggested by reviewer 2sdZ, we also provide a new run for BRO trained with L1 metrics computed from the internal embeddings of the same base model, circumventing the use of external models such as ESMC for data generation:

MethodT = 0.3T = 0.5T = 0.7T = 1.0T = 1.5Average
BRO-L1-self34.138.442.543.024.036.4

The results outperform Base/DPO/KTO and underperform BRO-L1 with ESMC embeddings. This suggests that an intrinsic L1 reward is also effective, but better representation model (since as opposed to the GPT-style progen2, ESMC was optimized for representation learning) would be more beneficial for post-training.

  1. Struct2Seq evaluation. Reviewer 4wq1 questioned whether our methods are applicable to Struct2Seq settings. We did perform Struct2Seq evaluation in Table 15 and 16, where BRO outperforms the base, inference-time scaling (best-of-n) and getting close to supervised baseline. We didn't discuss it extensively because according to reward diagnostics in Figure 1, pLDDT and pTM seems to be strictly better than our proposed heuristic metrics on Struct2Seq tasks, even though they are much more costly to evaluate (on a single A40 GPU, A40 GPU, the average inference time per sequence for ESMC-600M is less than 0.1s, whereas ESMfold takes more than 2.0s). However, this doesn't mean our proposed methods are not useful for Struct2Seq. As requested, we provide new Struct2Seq evaluations for ESM3-1.4B as follows (only 1 seed was run due to time constraint)

3D Coordinates conditional generation:

ModelMethodKeyword Recovery ↑
ESM3 (1.4B)Base0.197
ESM3 (1.4B)BRO-L10.255
ESM3 (1.4B)BRO-L1 Best-of-640.224
ESM3 (1.4B)Supervised0.260

Secondary structure (SS8) conditional generation:

MethodAccuracy ↑
Base0.719
BRO-L10.735
BRO-L1 Best-of-640.728
Supervised0.594

Solvent accessible surface area (SASA)conditional generation:

MethodSpearman ↑
Base0.746
BRO-L10.754
BRO-L1 Best-of-640.752
Supervised0.807

which demonstrates the effectiveness of our proposed method BRO-L1.

Overall, these results demonstrate that our proposed methods can be truly label-free, statistically superior across a variety of instrinsic and extrinsic proxy rewards, transfer/generalize to unseen prompt families and applicable to a variety of PLMs (autogressive ProGen2, multimodal masked language model ESM3) and conditional generation tasks (conditioning on function labels, 3D coordinates, secondary structure and solvent accessible surface area).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.