acceptodds
Under review as a conference paper at ICLR 2027

DP-DNA: Dual-Policy Reinforcement Learning for DNA Discrete Diffusion Models

Abstract

Reinforcement learning (RL) provides a powerful framework for steering generative models toward task-specific objectives and has shown promising results in regulatory DNA design. However, fine-tuning discrete diffusion models with downstream rewards can lead to undesirable distributional degeneration, where improved predicted activity is accompanied by reduced pattern diversity, distorted sequence statistics, and decreased biological naturalness. In this work, we first systematically characterize this failure mode and find that its severity is strongly generation-order dependent: after RL, different unmasking schedules produce markedly different degrees of sequence collapse. Motivated by this observation, we formulate discrete diffusion generation as two coupled decisions: and . We introduce a dual-policy RL framework DP-DNA, which optimizes both a generation-order planner and a content generator with a joint order–content GSPO objective. A free-generation stream trains both policies to generate DNA with high functional reward, while a real-sequence stream teaches them to reproduce the sequence patterns of real DNA. Across HepG2, K562, and SK-N-SH enhancer design, our method achieves high predicted activity while producing more natural sequence distributions, supporting dual-policy learning for function-directed DNA generation.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
DNA Design, Discrete Diffusion Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aL26OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies distributional collapse when using RL to optimize DNA discrete diffusion models. The main observation is that this collapse depends on generation order. Based on this, the paper proposes DP-DNA, which jointly learns where to generate and what to generate, together with a real-sequence training stream adapted from LoMDM. The results are generally strong across three enhancer-design tasks. My main concern is novelty and significance: most of the key ingredients already appear in DCoLT and LoMDM, so the new part is largely their combination in DNA RL. I am not yet convinced the current experiments isolate a large enough benefit from learned order itself to make the contribution sufficiently significant for ICLR.

Critical strengths
  • The paper identifies a meaningful failure mode beyond reward alone. Across both D3LM and SLM, RL improves predicted reward while substantially distorting GC content and local sequence diversity, and the reveal-order experiments show that these effects depend strongly on which positions are resolved first. This is a useful empirical observation for reward-based biological sequence design.

  • The empirical results are fairly strong. DP-DNA performs well across three cell types and multiple functional and distributional metrics, and the paper includes several component ablations rather than relying only on aggregate benchmark performance.

Critical weaknesses
  • Novelty: The methodological novelty appears limited relative to closely related work. DCoLT already jointly trains an unmasking policy and token generator with outcome-based RL, while LoMDM already learns context-dependent generation order using learned corruption and forward-reverse rate matching. DP-DNA combines these ideas in the DNA RL setting, but I am not yet convinced that this combination constitutes a sufficiently substantial methodological advance for ICLR.

  • Significance: The experiments do not cleanly isolate whether the gains come from adaptive generation order. The main D3LM+GSPO comparison changes several things at once, including the scheduler, advantage construction, and the additional real-sequence objective. The planner ablation is useful, but given how close the method is to prior work, a matched comparison varying only order learning is important for establishing the significance of the contribution.

Minor point: I also find the name DP-DNA confusing, since DP has a well-established meaning as differential privacy in ML. I would strongly suggest using a different acronym.

Questions for authors
  1. How does DP-DNA compare with a DCoLT-style joint order-content RL baseline using the same D3LM initialization, reward, regularization, and optimization budget?

  2. Can the authors provide a matched experiment where initialization, advantage construction, real-sequence supervision, and training budget are fixed, and only the learned planner is replaced by a fixed scheduler?

  3. The motivating order experiment changes which ground-truth tokens are revealed before completion. Does the same reward-collapse effect remain when starting from a fully masked sequence and changing only the generation-order policy while keeping the content model fixed?

4 comments

Dear Reviewer,

As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal.

We have addressed all your concerns about novelty and the contribution of learned generation order, including detailed methodological clarifications, matched planner ablations, DCoLT-style comparisons, and fully masked generation experiments.

We would also be happy to provide any further clarifications before the discussion closes.

Thank you for your time and consideration.

thanks for the revision and clarifications. i had also read through other reviews and author responses. i am not satisfied. the current draft does not pass the bar. I believe this paper would benefit from polishing more. so i keep my score.

also DP is prevalently known as Differential Privacy in the field of machine learning

Thank you for taking the time to review our revisions and responses. We appreciate your feedback and respect your assessment. If you have any further specific concerns or suggestions, we would be happy to address them.

Regarding the model name, as mentioned in our previous response, we agree that the acronym DP may cause confusion with Differential Privacy and plan to rename the model. We have kept the original name during the review process solely to maintain consistency and avoid confusion for reviewers when referring to the manuscript and our responses.

Thank you again for your time and consideration.

Reviewer_UogLOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies distributional collapse when RL fine-tunes discrete diffusion DNA models, reports that collapse severity depends on the generation (unmasking) order, and proposes DP-DNA, which jointly optimizes a position planner and a content generator with a shared GSPO objective plus a real-sequence stream that the paper describes as a realism-oriented regularizing signal.

Critical strengths

None

Critical weaknesses
  • The analysis in Section 3.2 and Figures 2–3 compares conditional completions from real-DNA contexts constructed using six reveal strategies, whereas the RL-tuned D3LM was trained with uniformly sampled generation positions. Structured strategies may induce partially revealed contexts that are underrepresented during RL training, making it difficult to distinguish increased order sensitivity from differential generalization across conditioning states. More importantly, at a fixed reveal count, the strategies expose different subsets of ground-truth tokens, thereby changing the information available to the decoder rather than isolating reveal order. This is particularly relevant for the oracle-based GT greedy and anti-greedy strategies, which explicitly select positions according to true-token predictability.
  • The planner ablations in Tables 2 and 6 do not report GC fraction or within-sequence 3-mer entropy, the two metrics central to the collapse analysis in Figures 1 and 6. The reported 3-mer correlation measures population-level compositional agreement and is not equivalent to within-sequence entropy. Furthermore, removing the real-sequence stream on K562 increases Pred-Activity (+0.359) and ATAC-Acc (+0.67 percentage points), while decreasing Evo 2 likelihood (−0.1275 nats/bp) and 3-mer correlation (−0.012), consistent with a realism-preserving regularization effect. Although the planner ablations show improvements in functional metrics and Evo 2 likelihood, the current evidence does not establish whether the reduced GC enrichment and preserved 3-mer entropy primarily arise from learned generation order, the LoMDM-inspired real-sequence objectives, or their interaction. Reporting the headline collapse metrics for the existing ablations, together with an inference-time uniform-order control, would make this attribution substantially clearer.
Questions for authors
  • Could the authors report GC fraction and 3-mer entropy for an ablation of the learned planner and the real-sequence stream, preferably at matched reward levels? Additionally, can reconstruction-only and rate-matching-only ablations clarify whether the diversity benefit comes primarily from content regularization or learned order?
  • Could the authors apply the Figure 2–3 diagnostic to DP-DNA checkpoints and compare the reward spread, GC fraction, and 3-mer entropy against vanilla RL? Does DP-DNA reduce order-dependent collapse, or primarily learn to operate in a more favorable region of the reward–realism trade-off?
1 comment

Dear Reviewer,

As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal.

We have addressed all your concerns through additional controlled experiments, including planner and real-sequence objective ablations, GC and H3 analyses, inference-time controls, and extended generation-order diagnostics.

We would also be happy to provide any further clarifications before the discussion closes.

Thank you for your time and consideration.

Reviewer_ehB1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper presents DP-DNA, a dual-policy reinforcement learning framework that jointly optimizes a context-dependent position-selection planner and a token content generator using a shared trajectory-level Group Sequence Policy Optimization (GSPO) objective. The study starts by investigating compositional distortion and reduced local-pattern diversity when fine-tuning discrete DNA diffusion models with RL. The authors observe that these effects depend strongly on the token unmasking order, which motivates learning generation order and content jointly. The proposed method combines a free-generation stream for reward optimization with a real-sequence stream for content reconstruction and forward–reverse rate matching. Across the Gosai MPRA enhancer-design benchmark for HepG2, K562, and SK-N-SH, DP-DNA achieves strong predicted regulatory activity, chromatin accessibility, and sequence-composition agreement, motif correlations, and Evo 2 likelihood relative to the evaluated baselines. Overall, I find the diagnostic analysis interesting and the approach well motivated, with strong empirical results. The following suggestions would strengthen the evaluation and clarify the method’s practical advantages.

Critical strengths
  1. The paper provides an insightful diagnostic analysis connecting token unmasking order to RL-induced compositional distortion, showing that generation order affects both predicted activity and local-pattern diversity.
  2. The combination of dual-policy RL with real-sequence reconstruction and rate matching is well motivated, giving both the planner and content generator complementary functional and real-data training signals.
  3. The method achieves strong empirical results across three enhancer-design tasks, improving predicted activity and several sequence-fidelity metrics relative to the evaluated baseline.
Critical weaknesses
  1. The evaluation would benefit from comparisons with GPA (Sarkar et al., 2026), DNA-CRAFT (Awasthi et al., 2026), and Ctrl-DNA (Chen et al., 2025). These methods provide relevant comparisons for reward-guided regulatory DNA generation and the trade-off between function and sequence quality. Comparisons should use common reward objectives and evaluation predictors, with computational costs reported to make the results interpretable.
  2. Assessing scalability to longer sequences. DP-DNA commits one token per generation step, so the number of sequential decisions grows with tokenized sequence length. Longer sequences also increase the cost of evaluating the Transformer at each step. The current experiments use only 200-base sequences represented by 35 tokens, and the paper does not establish how the method scales beyond this setting. Reporting generation time, training cost, and peak memory across sequence lengths would help clarify its applicability to longer regulatory elements.
  3. Benefit beyond tuned KL regularization. The comparison with D3LM+GSPO and the component ablations provide evidence for the proposed approach, but the baseline is evaluated at a single reported KL coefficient. It remains unclear whether tuning KL strength could recover a similar activity–sequence-quality trade-off. Furthermore, the use of KL regularization in the diagnostic experiments in section 3.1–3.2 is insufficiently specified. Clarifying those settings and including a KL-strength sweep would help establish what the planner and real-sequence stream add beyond conventional KL-regularized reward optimization.
  4. One suggestion would be to include broader evaluations. To strengthen evidence for generality, it would be useful to evaluate at least one additional regulatory-design benchmark beyond the Gosai enhancer tasks. Examples include an immune-cell chromatin-accessibility design task, as evaluated in DNA-CRAFT, or a promoter-design task, as evaluated in Ctrl-DNA.
Questions for authors
  1. What motivated the choice of non-overlapping six-mer tokenization? How dependent are the learned generation-order preferences and performance gains on this representation? Please discuss whether DP-DNA could be applied with single-nucleotide tokenization and the implications for computational cost and planning resolution.
  2. Do the RL models used in the collapse and generation-order diagnostics in section 3.1–3.2 include KL regularization? If so, please specify the reference model and coefficient. If not, could these diagnostics also be reported with KL regularization?
  3. Table 1 already reports batch-level 3-mer and JASPAR correlations. Could these metrics also be included in the diagnostic experiments, showing how agreement with high-activity natural sequences changes during RL training and across reveal strategies? This would connect the GC-fraction and within-sequence entropy observations to the distributional metrics used in the main evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.