High Reward, Wrong Sequences: Reward-Hacking Resistant Discrete Diffusion for 5’UTR Design

Reward-guided fine-tuning of discrete diffusion models offers a powerful route for de novo 5′UTR design, where the goal is to generate RNA sequences with high mean ribosome load (MRL) or translation efficiency (TE) while preserving biological plausibility. However, we find that this strategy is highly vulnerable to reward hacking: the predicted reward rises rapidly, while sequence naturalness, quantified by the statistical correlation between generated sequences and in-distribution natural sequences, collapses. We trace this failure to a simple but consequential mechanism: the reward model learns shortcuts that appear predictive on training sequences, then assigns inflated scores to unnatural sequences, pulling the generative policy away from biologically meaningful design space. We propose PILOT-UTR (Policy fIne-tuning with Learned-reward calibration and On-policy Teacher distillation for UTR design). PILOT-UTR first calibrates the reward model via using hacking policy to discover out-of-distribution sequences as label-free supervisions, suppressing inflated predictions on them through a cap loss. With this calibrated reward, PILOT-UTR then fine-tunes the generator through an ingenious teacher-student mechanism in which a slowly updated teacher acts as a moving biological anchor, guiding the student to improve reward while staying close to natural 5′UTR patterns. Our analysis explains how this teacher-student mechanism controls local prediction drift and gradually incorporates useful policy updates. Experiments on MRL- and TE-guided 5′UTR optimization demonstrate that PILOT-UTR mitigates reward hacking and achieves high predicted reward while preserving sequence naturalness and predicted structural stability. Code is available in https://anonymous.4open.science/r/PILOT-UTR-467F.