acceptodds
Under review as a conference paper at ICLR 2027

High Reward, Wrong Sequences: Reward-Hacking Resistant Discrete Diffusion for 5’UTR Design

Abstract

Reward-guided fine-tuning of discrete diffusion models offers a powerful route for de novo 5′UTR design, where the goal is to generate RNA sequences with high mean ribosome load (MRL) or translation efficiency (TE) while preserving biological plausibility. However, we find that this strategy is highly vulnerable to reward hacking: the predicted reward rises rapidly, while sequence naturalness, quantified by the statistical correlation between generated sequences and in-distribution natural sequences, collapses. We trace this failure to a simple but consequential mechanism: the reward model learns shortcuts that appear predictive on training sequences, then assigns inflated scores to unnatural sequences, pulling the generative policy away from biologically meaningful design space. We propose PILOT-UTR (Policy fIne-tuning with Learned-reward calibration and On-policy Teacher distillation for UTR design). PILOT-UTR first calibrates the reward model via using hacking policy to discover out-of-distribution sequences as label-free supervisions, suppressing inflated predictions on them through a cap loss. With this calibrated reward, PILOT-UTR then fine-tunes the generator through an ingenious teacher-student mechanism in which a slowly updated teacher acts as a moving biological anchor, guiding the student to improve reward while staying close to natural 5′UTR patterns. Our analysis explains how this teacher-student mechanism controls local prediction drift and gradually incorporates useful policy updates. Experiments on MRL- and TE-guided 5′UTR optimization demonstrate that PILOT-UTR mitigates reward hacking and achieves high predicted reward while preserving sequence naturalness and predicted structural stability. Code is available in https://anonymous.4open.science/r/PILOT-UTR-467F.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
RNA design, discrete diffusion, reinforcement learning, reward hacking, generative models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7GdVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces PILOT-UTR, which applies reward-finetuning to discrete diffusion for mRNA 5’UTR design, with calibrated rewards that claims to avoid reward hacking. The reward function is calibrated on OOD sequences generated with vanilla reward finetuning, and then used in finetuning with KL regularization. Experiments show improvements on independent rewards and metrics evaluating naturalism.

Critical strengths
  • PILOT-UTR does show evidence of preserving naturalness, with the highest 3-mer correlation values.
  • The paper aims to tackle reward hacking, which is an important problem in biological sequence design.
Critical weaknesses
  • The calibrated reward model makes assumptions on the reward on OOD sequences, whereas, the typical goal is to explore OOD with a generative model and then calibrate the reward model with true experimental data in an active learning loop. This method seems to be unsuited for further downstream applications where experimental validation is considered.
  • Scope on 5’UTR design is too narrow when the method can theoretically be applied to many modalities with hackable rewards, and relevant baselines/related works are not cited (e.g. [1], [2], [3], and [4]).

[1] Cao et al. (2025). GLIDE: A gradient-free lightweight fine-tune approach for discrete biological sequence design.

[2] Tang et al. (2025). Tr2-d2: Tree search guided trajectory-aware fine-tuning for discrete diffusion.

[3] Patel et al. (2026). mRNAutilus: Multi-Objective-Guided Discrete Generation of mRNA with Optimized Therapeutic Properties.

[4] De Santi et al. (2026). Active Flow Expansion for Out-of-Distribution Discovery: from Theory to Molecules.

Questions for authors
  • How does this framework apply to other modalities (e.g. proteins, peptides, small molecules, natural language?). Focusing on mRNA 5’UTR sequence seems too narrow a scope for a top ML conference venue like ICLR.
  • How does this compare to simply filtering out or performing negative finetuning on rollouts that result in OOD sequences during the finetuning process, rather than calibrating the reward?
Reviewer_fzteOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a reward-guided fine-tuning method for discrete diffusion models in 5′UTR sequence design. It uses EMA-teacher distillation to stabilize diffusion model fine-tuning. Experiments on MRL and TE design tasks show improvements.

Critical strengths

The paper addresses reward overoptimization in 5′UTR sequence design and provides empirical analyses of shortcut learning in reward models and the limitations of reward ensembles.

Critical weaknesses

First of all, I have some concerns about the paper's characterization of reward hacking:

In RL, reward hacking usually refers to a policy exploiting loopholes in the reward mechanism or discrepancies between a proxy reward and the true objective, achieving high rewards without actually fulfilling the intended goal.

However, what the paper calls "reward hacking" mainly arises from prediction errors in the learned reward model. This is not necessarily a problem with the reward objective itself, but rather with the accuracy and generalization of the predictor. Any selection, sampling, or optimization method relying on an imperfect scoring model can encounter the same issue. Therefore, I am not convinced that this should be characterized as a reward-hacking problem specific to RL.

Moreover, the actual objective of 5′UTR design is translation efficiency, not sequence naturalness or secondary-structure stability. Using reduced naturalness or structural stability as evidence of reward hacking is questionable.

  • First, these changes do not establish a discrepancy between the proxy reward and the true biological objective.
  • Second, weaker 5′UTR secondary structures may actually facilitate ribosome scanning and translation initiation, and lower 3-mer correlation may simply reflect exploration of novel sequence space. Therefore, these observations alone are insufficient to conclude that the generated RNA sequences are incorrect or that reward hacking has occurred.

Other concerns:

  • Prior work [1] has already studied a similar problem in 5′UTR design. Studies [2, 3] also investigated OOD reward overestimation and used conservative objective models to mitigate overoptimization. The idea of generating OOD samples through optimization and suppressing their predicted rewards is closely related to these prior approaches.

  • Policy distillation has also been explored for stabilizing reward-guided diffusion fine-tuning in biomolecular design [5].

  • The theoretical analysis mainly follows from Pinsker's inequality and standard EMA properties, without establishing any guarantee against reward hacking.

  • Table 7 shows fairly similar performance between the frozen and EMA teachers. How much does the adaptive teacher mechanism actually contribute?

  • The reward calibration procedure: The method imposes a prediction cap on high-scoring OOD sequences without knowing their true functional values. However, OOD sequences do not necessarily have low MRL or TE and may include genuinely effective novel designs. Why should their predicted rewards be suppressed?

  • Table 5 shows that recalibration worsens both validation MSE and Pearson correlation on the Human MRL and PC3 TE tasks. Does this procedure actually correct reward-model errors, or does it simply suppress predictions on OOD sequences?

  • The criteria used to identify “reward hacking” are not well justified. The authors mainly rely on reduced 3-mer correlation and MFE approaching zero, and Table 4 uses manually chosen thresholds to classify reward hacking. However, these metrics do not directly measure translation efficiency. Weaker secondary structures in 5′UTRs can sometimes facilitate ribosome scanning and translation initiation. Why should reduced structural stability necessarily indicate a failed design?

[1] Sample et al., Human 5′ UTR Design and Variant Effect Prediction from a Massively Parallel Translation Assay, 2019.

[2] Trabucco et al., Conservative Objective Models for Effective Offline Model-Based Optimization, ICML.

[3] Uehara et al., Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Models, NeurIPS.

[4] Tarvainen et al., Mean Teachers Are Better Role Models: Weight-Averaged Consistency Targets Improve Semi-Supervised Deep Learning Results, NeurIPS.

[5] Su et al., Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Design.

Questions for authors

Please see the "Critical Weaknesses" part.

Reviewer_X2zoOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work presents PILOT-UTR, a reward-guided fine-tuning framework that aims to mitigate reward hacking in discrete diffusion models for UTR design. The authors use a weakly regularized policy to generate OOD sequences for reward-model calibration through a cap loss, then fine-tune the generator using the calibrated reward and on-policy EMA-teacher distillation. They evaluate the framework on three benchmarks targeting MRL and TE, using independent evaluators and metrics of sequence naturalness and predicted structural stability.

Critical strengths
  1. The framework is straightforward, combining reward-model calibration with policy stabilization. Using policy-generated sequences for calibration avoids the need for additional experimental labels.
Critical weaknesses
  1. I'm concerned that the current evidence can't confirm reward hacking, or that it is simply a trade-off between objectives. Figure 1 shows an increase in predicted reward, but a deterioration in compositional and structural indicators. How can we support the claim that "the gap between proxy reward and true biological function is widening rapidly"? In particular, why is a near-zero MFE or a decrease in 3-mer correlation sufficient to determine hacking, rather than simply a trade-off between functional optimization and distributional similarity? I suggest adding the independent-evaluator (MRL) curve from Figure 7(a) to Figure 1.
  2. It remains unclear whether PILOT-UTR outperforms adequately tuned DRAKES. The limited α\alpha sweep does not establish a convincing reward–fidelity trade-off. Please sweep the regularization strength densely and plot full reward–fidelity Pareto fronts for both methods.
  3. What direct evidence supports attributing failure to composition-based shortcuts? Sec. 3 infers from metric degradation that the reward model uses spurious features, but other explanations don't seem ruled out. Can you present the feature distributions for the training data, high-function subsets, and generated data, and how these features relate to measured MRL/TE? How can we distinguish between "non-causal shortcuts" and "functionally meaningful features being incorrectly extrapolated"? Are zsz_s and zgz_g conceptual assumptions, or have we identified them experimentally? A feature-ablation or intervention test on the reward model would make the shortcut claim causal.
  4. The loss relies on an insufficiently validated assumption that policy-generated OOD sequences should receive scores below a prescribed threshold. Without labels, this procedure may suppress useful OOD candidates. Please validate the threshold on held-out labeled sequences, and report sensitivity to its value.
Questions for authors
  1. Figure 1 reveals that reward hacking occurs by seeking high-reward samples. If we can get these metrics without the expense, why not add them as regularization to the DRAKE loss function?
  2. How do you verify that the "hacked policy" successfully exploits the reward model before collecting calibration samples? How diverse are these samples, and can newly optimized policies still exploit the calibrated reward model?
  3. How does simple ensemble averaging compare with the uncertainty-penalized objective in Appendix D.5? Would changing the uncertainty penalty or increasing ensemble diversity affect the observed failure?
  4. Please specify the α\alpha, checkpoint, and example-selection procedure used in Figure 2. For Figure 5, could you provide comparisons using a common reward evaluator and comparable optimization budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.