- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper mainly discusses using model-generated proxy rewards to perform offline post-training (e.g. offline RL) on protein large language models, thereby improving conditional generation performance without requiring task-level ground-truth reward labels for generated samples (which is too slow). The authors propose offline algorithms such as SRO and BRO and conduct experiments across various tasks. The results demonstrate the effectiveness of the proposed methods, with additional results provided in the appendix.
Critical strengths
I find the research on offline RL for PLMs to be quite significant; the authors analyzed the relationship between surrogate rewards and temperature and observed some interesting results/findings, which I consider great This work offers valuable insights into reward selection/design and data sampling.
In general, the proposed method demonstrates effectiveness across few models, and the authors present ablation results for the various components.
Critical weaknesses
Although the paper presents some experiments, there remain significant concerns. The current evaluation seems insufficient to demonstrate the broad applicability of the proposed method for the offline post-training of PLMs, as the tasks addressed appear somewhat limited. Furthermore, the paper fails to show whether the method achieves reliable generalization across a wide range of unseen tasks. I encourage the authors to address these problems in their discussion.
Additionally, I am quiet curious whether the proxy reward relies on final labels (e.g., ground-truth, real label), or if the proposed method can be applied to tasks where ground-truth labels are unavailable. I suggest the authors provide a detailed discussion of the method and experimental setup.
Finally, given the use of a reward proxy, why did the authors not compare their approach against mainstream online RL algorithms, such as GRPO?
Questions for authors
See Critical Weaknesses and,
I recommend that the authors address issues regarding the applicability of the proposed method, its reliance on ground-truth (GT) labels, and the lack of sufficient baselines for offline and online RL.
Rebuttal
We sincerely thank the reviewer for the constructive review and for identifying some of the key strengths of our paper. For your concerns, we respond point by point. Please also refer to the seperate General Response thread on top for additional experiments and discussion of commonly raised questions.
W1: Applicability of the proposed method
Since reviewer SDDq didn't specify the limitation, we can only audit on the scope of applicability that has been covered in our experiments:
Additionally, in General Response on top, we have demonstrated positive transfer on heldout family and applicability with base-model-generated embeddings. Overall, we think these results have demonstrated broad applicability of our proposed method.
W2: Reliance on ground-truth (GT) labels
We appreciate the concern. However, we must clarify that our benchmarking results in section 4.2 are merely diagnostics to showcase the correlation between proxy rewards and the actual ground truth labels, i.e., to evaluate if these metrics qualify as a proxy of the ground truth labels. This procedure is not required and can be skipped entirely for post-training purpose. As shown in Table 2, simply using the L1 metric across all temperatures achieves 0.433 (BRO) and 0.389 (SRO) in keyword recovery, significantly enhancing the base performance (0.281) and outperforming DPO (0.350) and KTO (0.337) under the same setting. We also provide new results on multi-seed runs (see General Response) to show that the gains are statistically significant. Given that L1 metric is task- and temperature-agnostic by requiring only a representation model to generate sequence embeddings, our proposed post-training method can achieve significant and meaningful gains without any information about the actual ground truth. Table 2 also shows that other simple metrics such as model likelyhood also bring notable gains (BRO 0.341 and SRO 0.364 vs. Base 0.281) without ground truth information, so L1 is not just a lucky or cherry-pick case. Therefore, metrics like L1 and likelihood can be used and effectively improve the base model in a zero-shot, offline setting.
So the key is that post-training procedure is independent of the reward diagnostics, despite the fact that we present it as section 4.3, right after the reward benchmarking as section 4.2. We will adjust our writing in the final version accordingly to make it clear.
W3: Lack of sufficient baselines for offline and online RL
For offline RL, we have included DPO and KTO, which are competitive baselines widely applied for LLMs and protein lanaguages models. These methods are most relevant because they use binary labels and do not require continuous, explicit rewards, just like BRO. We restrict our method and evaluation to offline setting because 1. It is more practical: in protein engineering and biology, there are many unlabeled datasets that could be used, while the labeling is the bottleneck. Also given that a typical protein experiment takes days or even weeks, it's impossible to train a PLM by interactively receiving online feedback from the experiments. 2. For online training with our proposed unsupervised rewards, which can be evaluated much faster, there is the risk of overoptimization: e.g., He et al [1] observed consistent reward hacking and model collapse at later stages when using intrinsic rewards such as model entropies and likelihood. Therefore, in this paper we prioritize offline RL setting and leave online development for future work.
[1] He, Bingxiang, et al. "How Far Can Unsupervised RLVR Scale LLM Training?." International Conference on Learning Representations. Vol. 2026. 2026.
Hope our reply addresses your concerns and clarifies any misunderstanding. If you find them helpful, please consider raising your score, thank you!