- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper presents Ariadne, a reinforcement-learning framework for regulatory DNA design. It first fine-tunes an autoregressive DNA language model on MPRA-derived sequences to establish a shared, unconditional sequence prior, then trains separate target-cell policies using Enformer-guided PPO. Its main contribution is an adaptive reward combining EMA-based activity normalization, within-batch ranking, and a bounded bonus for exceeding a historical upper-tail threshold.
Validation comprises computational experiments in HepG2, K562, and SK-N-SH, evaluated using both the training reward modelan external predictor. Reward ablations and analyses of training dynamics assess the contributions of individual reward components and their effects on predicted activity and sequence diversity.
Critical strengths
-
The paper clearly motivates the challenges in regulatory DNA design and presents the proposed method in a well-organized manner. Its explanation of the two-stage training pipeline and the roles of individual reward components makes the approach easy to understand and assess.
-
The appendices provide detailed hyperparameters, training algorithms, dataset splits, and evaluation procedures, including the reward-state updates and stagewise PPO settings. These implementation details improve transparency and facilitate reproduction of the proposed method.
Critical weaknesses
- The paper has insufficient demonstration that the reward design addresses the motivating problems. Although the intended roles of EMA, rank, and tail are explained, the main paper does not directly characterize the changing reward distributions or demonstrate how the proposed components address the resulting optimization difficulties.
- The paper has insufficient controlled comparison with existing RL methods. The paper lacks protocol-matched comparisons with alternative algorithms such as GRPO, while the TACO results come from a different training protocol. Comparisons using the same initialization, oracle, and query budget are needed to establish whether Ariadne offers a meaningful improvement over existing RL-based design approaches.
Questions for authors
- Is the proposed reward intended primarily to accommodate changing reward scales, or also to mitigate concentration of generated sequences? Table 1 reports Hamming diversity, while the appendix reports novelty and representation concentration. What do these results establish about the trade-off between predicted activity and sequence quality?
- Ariadne already incorporates adaptive KL regularization, a mechanism used in prior RL fine-tuning work to limit reward-model overoptimization [1]. What evidence separates the benefits of the proposed shaping from changes in the effective balance between activity reward and KL regularization? Prior work highlights the importance of controlling implementation details when attributing RL performance gains [2].
- How does the proposed method differ from established reward scaling and clipping [2], relative-ranking reward designs [3], and upper-quantile, risk-seeking policy optimization [4]? Is the methodological contribution primarily the particular combination of EMA normalization, ranking, and a historical tail threshold, or a distinct mechanism?
[1] Ouyang et al. Training Language Models to Follow Instructions with Human Feedback. NeurIPS, 2022.
[2] Engstrom et al. Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO. ICLR, 2020.
[3] Niu et al. From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning. ICML, 2026.
[4] Petersen et al. Deep Symbolic Regression: Recovering Mathematical Expressions from Data via Risk-Seeking Policy Gradients. ICLR, 2021.
5 comments
Predicted activity and cross-predictor agreement
Fixed-candidate results. The table reports Enformer top-16 (E) and Malinois on those same sequences (M), as mean ± standard deviation over five training seeds. Top-16 is computed within each 256-sequence draw and averaged over 59 generation seeds per policy.
| Reward | HepG2 E / M | K562 E / M | SK-N-SH E / M |
|---|---|---|---|
| EMA | 0.912±0.023 / 0.819±0.020 | 0.990±0.006 / 0.765±0.059 | 0.923±0.033 / 0.897±0.041 |
| EMA+rank | 0.928±0.018 / 0.834±0.031 | 0.995±0.007 / 0.781±0.066 | 0.958±0.019 / 0.895±0.021 |
| EMA+tail | 0.888±0.098 / 0.792±0.075 | 0.995±0.005 / 0.785±0.050 | 0.909±0.086 / 0.841±0.140 |
| ERT | 0.933±0.021 / 0.794±0.054 | 0.993±0.006 / 0.774±0.056 | 0.940±0.034 / 0.910±0.014 |
Sequence quality. Exact uniqueness is 1.0 in each merged pool of 15,104 sequences. For the pooled Enformer top-128, median pairwise Hamming distance is 129.8–137.0 bp for EMA and 128.0–137.6 bp for ERT across cell types. Their mean pairwise Enformer-embedding cosines are 0.974–0.984 and 0.980–0.987, respectively. Sequence dispersion therefore coexists with embedding similarity; this alone does not prove oracle exploitation.
Prior work and GRPO. Our method combines historical normalization/clipping, within-batch ranking, and a bounded bonus above a historical threshold. These relate to established scaling, relative-reward, and upper-tail methods; we do not claim a new policy-gradient estimator.
PPO-ERT and GRPO-ERT share initialization, oracle, stagewise reference schedule, and scheduled update count. They differ in KL coefficient (0.12 vs. 0.04), whitening, and advantage estimation. GRPO can skip minibatches, and actual update counts are unavailable; therefore, this is an exploratory comparison.
| Method | HepG2 E / M | K562 E / M | SK-N-SH E / M |
|---|---|---|---|
| PPO-ERT | 0.933±0.021 / 0.794±0.054 | 0.993±0.006 / 0.774±0.056 | 0.940±0.034 / 0.910±0.014 |
| GRPO-ERT | 0.876±0.027 / 0.844±0.076 | 0.967±0.008 / 0.785±0.059 | 0.975±0.017 / 0.865±0.046 |
Neither method is better on both predictors across all cell types. GRPO-ERT has higher Enformer–Malinois Spearman correlation on all five HepG2 and K562 training seeds: 0.641±0.045 vs. 0.098±0.083 for HepG2, and 0.580±0.147 vs. 0.170±0.075 for K562 (GRPO-ERT vs. PPO-ERT). Top-128 overlap is also higher for GRPO-ERT in these two cell types. These results indicate stronger cross-predictor ranking agreement in these settings.
Sequence diversity and quality
Sequence diversity and quality. We computed Hamming diversity and embedding similarity on the Enformer-selected top-128 sequences from each merged pool, and computed sequence violations on the full merged pool of 15,104 sequences per training seed. The table reports median pairwise Hamming distance (bp), summarized across five training seeds. Paired differences are PPO-ERT minus GRPO-ERT.
| Cell type | PPO-ERT | GRPO-ERT | Paired difference | PPO higher |
|---|---|---|---|---|
| HepG2 | 137.6±3.4 | 134.6±2.2 | +3.0±3.9 | 3/5 seeds |
| K562 | 128.0±4.4 | 118.8±2.8 | +9.2±6.4 | 5/5 seeds |
| SK-N-SH | 131.8±8.8 | 111.8±32.5 | +20.0±33.1 | 4/5 seeds |
The SK-N-SH mean difference is strongly influenced by one GRPO-ERT seed with Hamming distance 54. On HepG2, the merged-pool difference is positive in only three seeds. When diversity is instead computed within each 256-sequence generation draw and averaged, PPO-ERT is higher in 5/5, 5/5, and 4/5 seeds for HepG2, K562, and SK-N-SH, respectively.
Other metrics do not show a uniform PPO advantage. For the Enformer-selected top-128 from each merged pool, mean pairwise Enformer-embedding cosine is 0.981±0.004 vs. 0.979±0.003 in HepG2, 0.987 for both methods in K562, and 0.980±0.007 vs. 0.994±0.005 in SK-N-SH (PPO-ERT vs. GRPO-ERT). Novelty is similar in HepG2 and approximately 2 bp higher for PPO-ERT in K562 and SK-N-SH. Merged-pool uniqueness is 1.0 for both methods except GRPO-ERT SK-N-SH seed 2 (0.942).
In the full merged pool of 15,104 sequences per training seed, the sequence-violation fraction is lower for PPO-ERT on all five K562 and SK-N-SH seeds: 0.045±0.049 vs. 0.738±0.096, and 0.144±0.107 vs. 0.896±0.142, respectively. A sequence is counted as violating if its GC fraction is outside 0.25–0.75, it contains an invalid base, or its homopolymer length exceeds 6. This is a reporting metric, not a training objective. All comparisons are descriptive and do not isolate the effect of the RL algorithm.
Reward attribution and training protocol
We thank the reviewer for asking us to distinguish reward shaping from scale, KL regularization, and prior methods. We audited the implementation and clarify the following.
Training protocol. All PPO configurations start from the same SFT checkpoint and use ten stages of 100 scheduled steps. Stages 2–10 start from the preceding stage's Enformer-selected best policy. We report the stage-10 final policy, without early stopping or selection across seeds. Thus, training has a fixed schedule but includes stagewise Enformer-based checkpoint selection.
Reward and KL. All configurations use reward whitening and a fixed KL coefficient of 0.12. Terminal PPO approximate KL values are broadly similar, but KL trajectories and applied update counts are unavailable. We cannot fully separate reward shaping from effective optimization strength.
EMA is a historical z-score clipped at ±5 and reset at each stage. Without clipping, per-rollout whitening cancels this affine transformation in exact arithmetic. Since clipping frequency was not logged, these runs do not establish a benefit from historical EMA. The original adaptive-KL protocol did not use this whitening and must be interpreted separately.
Interpretation. The reward acts on oracle scores; it does not explicitly prevent representation concentration. Full ERT does not consistently outperform simpler rewards across cell types and predictors. We will narrow the contribution to an empirical study of reward configurations and their limitations.