acceptodds
Under review as a conference paper at ICLR 2027

TC-NMD: Trajectory-Conditioned Non-Markovian Diffusion for DNA Sequence Design

Abstract

Discrete diffusion models have emerged as a competitive framework for biolog- ical sequence design, achieving strong results on DNA enhancer and promoter generation benchmarks. Yet the models driving these results, masked diffusion, Dirichlet flow matching, and the simplex-based shortlisting model (SLM), all share a common structural assumption: the reverse process is Markov, so each denoising step conditions only on the current noisy state. This is convenient, it yields tractable per-step losses and ancestral sampling, but it means the reverse step ignores the rest of the sampling trajectory, even though those noisier states are already computed during generation. Recent non-Markovian discrete diffusion models show that conditioning the reverse process on this trajectory improves generation quality. We propose TC-SLM, a non-Markovian simplex diffusion model that brings this idea to DNA sequence design. TC-SLM represents corrupted DNA as simplex-valued distributions over nucleotides and, at each reverse step, augments the denoiser with a learned summary of the cached noisier states: a Gumbel-Softmax router scores those states by their compatibility with the current state and forms a routed history context used to predict the clean sequence. On conditional enhancer and promoter design, and on unconditional protein genera- tion, a non-Markovian model trained from scratch improves over its Markovian counterpart, most substantially on fly-brain enhancer design, while adding only a lightweight router to the architecture.

Primary area:
generative models
Keywords:
Diffusion, Discrete diffusion, DNA sequence design

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_ukxdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TC-NMD, a non-Markovian discrete diffusion framework for biological sequence design. It combines an independent-corruption forward process, a learned trajectory summary, and a biased corruption schedule so that intermediate decisions can be revisited. The framework is instantiated with both TC-SLM and TC-MDLM. With 5 sampling steps, the proposed models outperform their corresponding Markovian baselines on enhancer and promoter design tasks.

Critical strengths
  1. The method is well motivated by a clearly identified problem. It preserves the marginal distribution at each noise level while changing the joint distribution over the trajectory. The same framework can also be applied to different backbones, including SLM and MDLM, suggesting a degree of generality.

  2. The few-step experimental evaluation is relatively broad. The paper compares against the main experimental settings reported in the SLM paper and additionally includes motif-profile experiments.

  3. The router is lightweight. It introduces few additional parameters and does not substantially increase the computational cost.

Critical weaknesses
  1. Limited multi-step evaluation. Prior work generally reports performance in a multi-step regime, often using around 100 NFE, whereas this paper mainly compares TC-NMD and its reproduced baselines at 5 NFE. Consequently, the paper does not establish whether the observed gains persist when more sampling steps are allowed. Results across a broader range of NFE values, including 100 NFE, would make the comparison more complete.

  2. Dataset- and backbone-specific tuning may impose a large inference-time tuning cost. The bias bb and the classifier-free guidance scale are selected separately for each dataset and backbone, and their optimal values appear to vary considerably across tasks and architectures. In addition, obtaining the best inference performance requires searching over the guidance weight ww, which may result in a substantial hyperparameter selection budget.

  3. Computational-cost measurements are mainly reported for the SLM backbone. The paper provides detailed measurements for SLM, but gives limited or no corresponding information for the MDLM backbone. It is therefore unclear whether the claimed computational efficiency and memory overhead generalize to MDLM.

  4. Minor: incorrect table references remain in the manuscript. Several references still contain placeholders such as “Table ??,” which reduces the clarity and polish of the presentation.

  5. Minor: the legend in Fig. 4 is somewhat confusing. The melanoma curves are shown with dashed lines, but this line-style convention is not clearly explained in the legend. The figure remains interpretable, but the legend should explicitly identify the meaning of the dashed and solid lines.

Questions for authors
  1. Is the simplex superscript in Eq. 10 intended to be ΔT−t−1\Delta^{T-t-1}? Since πϕ\pi_\phi outputs a probability distribution over τ\tau, there are T−tT-t probability components and therefore T−t−1T-t-1 degrees of freedom under the usual geometric definition of a simplex.

  2. Could the authors report the performance of TC-NMD and the SLM/MDLM baselines across a wider range of sampling steps, including 100 NFE? The SLM baseline performs substantially better at 100 NFE than at 5 NFE. If TC-NMD does not retain a clear advantage in the multi-step regime, could the authors explain why its benefits are primarily observed in few-step sampling and clarify the intended application regime?

  3. Could the authors specify precisely how ctc_t is computed in the experiments in Section 3? Does this quantity require additional hyperparameter tuning? Please also report whether the experimental results are robust to different choices or parameterizations of ctc_t, ideally through a sensitivity analysis or ablation study.

5 comments
AuthorsOpenReview ↗

Response to Reviewer ukxd part 1

We thank the reviewer for recognizing the motivation of our method, its applicability to different backbones, and its improvements in the few-step sampling regime. We address the reviewer’s concerns below:

Q1 — Simplex notation

We thank the reviewer for pointing this out. The reviewer is correct that the router outputs a probability distribution over T−tT-t history states, and therefore the simplex has T−t−1T-t-1 degrees of freedom. We will correct the notation in Equation 10 to ΔT−t−1\Delta^{T-t-1}.

Q2, W1 — Performance at higher NFEs

We thank the reviewer for raising this concern. We agree that our method is most useful in the low-sampling regime, and we will make this clear in the revised paper and discuss its performance at higher NFEs.

At 5 NFE, TC-SLM substantially improves FBD on Fly Brain (14.65 → 2.02 conditional, 19.57 → 4.79 unconditional) and Melanoma (9.95 → 4.64 conditional, 10.11 → 5.74 unconditional). To further demonstrate its practical efficiency, we measured generation time on an H100 GPU (batch size 64, sequence length 500). On conditional Fly Brain, SLM requires approximately 17 NFE to approach the quality of TC-SLM at 5 NFE. Generating 8,192 sequences takes 9.0 seconds with TC-SLM versus an estimated 28.2 seconds with SLM, giving approximately 3.1× faster generation, including the computational overhead of our method. For one million sequences, this corresponds to approximately 18.4 versus 57.4 minutes, extrapolated from measured throughput.

We also investigated performance at higher NFEs and evaluated an adjusted version of TC-SLM on Melanoma. The following table reports FBD (conditional / unconditional), with all adjustments selected on the validation split and results averaged over three seeds.

NFETC-SLM (adjusted)Without re-maskingSLM
54.64 / 5.746.69 / 6.959.95 / 10.11
102.99 / 4.144.20 / 5.574.39 / 5.96
202.13 / 3.723.66 / 4.952.11 / 3.55
502.63 / 3.613.18 / 4.641.33 / 2.94
1002.66 / 3.593.48 / 4.391.21 / 2.82

Values are conditional / unconditional means; SDs are omitted for brevity. The adjusted method uses validation-selected re-masking budget (kk) and router temperature (τ\tau).

At higher NFEs, applying re-masking at every step can repeatedly reintroduce excluded candidates and prevent progressive refinement. We therefore limit re-masking to a fixed budget (k=16k=16 at 20–100 NFE), while also adjusting the router temperature where needed. The results show that re-masking remains beneficial compared with removing it entirely in high NFE settings.

At very high NFEs, SLM still performs better. One possible reason is that the router must select useful information from a much larger number of history states (up to 99 at 100 NFE, compared with 4 at 5 NFE). More advanced router architectures may improve performance in this setting, which we leave for future work.

Overall, these results support our main motivation of improving generation quality with fewer sampling steps. This is particularly useful for large-scale biological sequence design, where many candidates are generated and screened before experimental testing.

For example, Bennett et al. (2023) generated approximately 2 million protein designs per target and selected around 20,000 for experimental testing, explicitly considering computational efficiency in their design pipeline. Dai et al. (2026) further demonstrated that increasing the number of sampled protein candidates can improve the predicted quality of selected designs. These studies demonstrate the practical importance of efficient sampling in biological sequence design. We will include these additional experiments and clarify the low-sampling focus and higher-NFE limitations in the revised paper.

Bennett et al. Improving de novo protein binder design with deep learning.

Dai et al. Programmable design of functional proteins from natural language.

AuthorsOpenReview ↗

Response to Reviewer ukxd part 2

W2:

We thank the reviewer for raising this concern. The inference-time hyperparameter tuning cost is relatively small. One validation FBD evaluation at 5 NFE takes approximately 15–30 seconds on a single H100/H200 GPU for both TC-SLM and TC-MDLM. Therefore, evaluating nine bias values takes around 3–5 minutes per setting. Our complete TC-MDLM hyperparameter sweep over 7 bias and 5 guidance values took approximately 15 minutes on an A100 GPU. We also found that the selected values are within a relatively narrow range: 0–0.07 for TC-SLM bias, 0.15–0.25 for TC-MDLM bias, and a guidance scale of 1 for all MDLM settings.

We also noticed that the guidance weights reported for the MDLM baselines had been tuned on an earlier version of our code, and these values were carried over after changes to the implementation. We have now repeated the tuning using the final version. On conditional Fly Brain, the MDLM baseline improved from 9.79 to 6.15, while TC-MDLM improved from 6.07 to 2.53 after tuning the inference bias. On conditional Melanoma, the baseline improved from 14.34 to 7.44, while TC-MDLM remained at 6.48. The updated results show that TC-MDLM still achieves lower FBD than MDLM on both datasets. On Fly Brain, TC-MDLM achieves 2.43× lower FBD, and the absolute improvement remains almost the same as in the original results. On Melanoma, the improvement is more modest after retuning (7.44 → 6.48, a 12.9% reduction). We will update these results in the revised paper and make a separate public comment on OpenReview to report the corrected comparisons to all reviewers and the area chair.

Q3:

Regarding the history weights, ctc_t in Equation 12 is the corruption level obtained directly from the forward noise schedule at timestep tt. We use wcur=1+ctw_{\mathrm{cur}}=1+c_t and whist=1−ctw_{\mathrm{hist}}=1-c_t, so these weights do not require any additional hyperparameter tuning.

We also conducted an additional ablation using the trained TC-SLM checkpoints with different fixed weights. The following table reports FBD at 5 NFE using the best checkpoint for each setting.

Current / History weightsMelanoma Cond.Melanoma Uncond.Fly Brain Cond.Fly Brain Uncond.
1+c1+c / 1−c1-c (ours)4.715.311.614.64
1.5 / 0.54.765.391.614.84
1.2 / 0.84.615.321.514.38
1.0 / 1.04.665.441.594.32
0.8 / 1.24.685.481.604.05
0.5 / 1.55.065.912.485.06
2.0 / 0.0 (no history)30.4233.6223.0129.87

The results show that our method is not highly sensitive to the exact choice of history weights within a reasonable range. Using different fixed weights gives similar results to our noise-dependent weighting, and in some cases slightly better results. However, removing the history contribution substantially increases FBD in all settings. These results show that the method benefits from the trajectory history without requiring careful tuning of its contribution.

W3:

Regarding the computational cost for MDLM, we conducted additional measurements using the same experimental setup as Table 6. The results are reported below.

MetricMDLMTC-MDLMRelative
Forward MACs (GMAC)95.0895.211.0013×
Training time (ms/step)20.0922.751.13×
Inference time (ms/batch)66.9073.451.10×
Training peak memory (MB)1313.741804.061.37×
Inference peak memory at 5 NFE (MB)338.48343.091.014×

The results show that the computational overhead is similar to the SLM backbone. TC-MDLM adds only 0.07% additional parameters and increases the forward MACs by approximately 0.13%. The inference-time overhead is approximately 10%, while the peak inference memory increases by only 1.4% at 5 NFE. The training memory overhead is higher because the model uses multiple noisy views during training.

We hope that the additional experiments and clarifications adequately address the reviewer's concerns and provide stronger evidence for the contributions of our method. We would appreciate it if the reviewer could reconsider their rating.

Minor issues:

We will address the missing placeholders, and clarify the legend in figure 4.

Dear Reviewer,

Thank you again for your time and feedback. We have addressed your concerns in our rebuttal and updated the manuscript with additional experiments on higher sampling steps, computational cost, and further clarifications.

We would appreciate it if you could review our responses and let us know if you have any remaining concerns. We would be happy to discuss them further.

Thank you for your time and consideration.

Reviewer_w1rUOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses issues arriving from the Markovian assumption in the diffusion reverse process: namely that this assumption prevents decoded tokens or excluded candidates from being reconsidered. The paper presents a new method to include history of past states in the prediction of the state at t+1. The author's method shows strong improvement from the previous main method (SLM), which this method builds upon.

Critical strengths

This paper successfully addresses a limitation of existing approaches and shows quite strong empirical results compared against an appropriate range of different diffusion model styles.

The paper also includes convincing ablations suggesting that the history introduced by their method is what is driving improvements.

Minor note: the appendix is largely well organized and useful

Critical weaknesses

Their theoretical contributions are not well introduced. Part of that could be do to the way that this information was presented (I have included some examples of this below). Overall I do not believe that this paper makes any major theoretical contributions, but still seems to be a useful method.

Nits:

Line 47: “Function depends on local motifs and their arrangement” - function is not only dependent on local motifs, distant motifs can also have interactions.

Around line 86 notation is used that has not been introduced

FBD should be defined upon first use in the Intro

I might be missing something, but it looks like aside from colors the diagram 3b markov corruption is the same as 3b independent corruption.

Mis-capitalization “FLyBrain dataset” on 446

Questions for authors

I am curious about the robustness of this method to higher dimensional vocabularies. Sometimes methods show instabilities when used for protein or language data, that do not reveal themselves for vocab size 4 DNA data.

One thing that would make these results even stronger would be to show performance on a larger vocabulary domain such as proteins.

3 comments
AuthorsOpenReview ↗

Response to Reviewer w1rU

We thank the reviewer for acknowledging the motivation of our method, the empirical improvements, and the supporting ablation studies. We address the reviewer’s concerns below:

Q, W1) the applicability of our method to larger vocabularies:

We thank the reviewer for this suggestion. To examine the applicability of our method to larger vocabularies, we conducted a preliminary experiment on protein sequence generation using UniRef50. We implemented TC-SLM using the same DiT backbone as the SLM protein model. The protein model uses a vocabulary of 31 tokens, including 20 canonical amino acids, compared with the vocabulary size of 4 for DNA.

We evaluated the generated sequences using ESM-2 perplexity and ProGen2 NLL (lower is better). The following table reports the results at 5 and 10 NFE, along with amino-acid entropy to examine whether lower perplexity is caused by reduced sequence diversity.

NFEModelESM-2 PPL ↓ProGen2 NLL ↓Entropy ↑
5SLM16.0182.8263.780
5TC-SLM15.2522.7833.855
10SLM15.8652.8263.858
10TC-SLM15.3152.7853.861

The results show that TC-SLM achieves lower perplexity on both evaluators at 5 and 10 NFE, while maintaining comparable or higher amino-acid entropy.

We also did not observe duplicate sequences or mode collapse in the evaluated samples.

However, the Fréchet Protein Distance (FPD) does not show the same improvement. At 5 NFE, SLM achieves an FPD of 10.5, compared with 13.9 for TC-SLM. These preliminary results show that our method can be applied to a larger vocabulary and a transformer-based denoiser without any instabilities.

W2) typos: We thank the reviewer for pointing out these issues. We will revise line 47 to clarify that regulatory function can also depend on interactions between distant motifs, introduce the notation before its first use, define FBD in the introduction, and correct the capitalization in line 446.

W2) Figure 3(b): In the Markovian corruption process, the candidate set at time tt is a subset of the candidate set at time t+1t+1. However, in the non-Markovian corruption process, this subset relationship does not necessarily hold, and the previous states can provide additional information about the clean sequence. For example, in Figure 3(b), taking the intersection of the two independently corrupted candidate sets recovers the true token, while in the Markovian case, the intersection only gives the smaller candidate set.

Regarding “Their theoretical contributions are not well introduced.”: Our main methodological contribution is a non-Markovian diffusion framework that uses information from previous states and allows intermediate decisions to be revised. We also provide theoretical support by deriving a variational lower bound for the non-Markovian formulation and proving that its per-step KL terms are finite. We demonstrate the effectiveness of our method across SLM and MDLM, supported by empirical improvements and ablation studies. We will clarify these contributions in the revised paper.

We hope that the additional protein experiments and clarifications address the remaining concerns and strengthen the reviewer's assessment of our work.

Reviewer_w1rUOpenReview ↗

Concerns addressed- score updated

Dear Authors, Thank you for addressing my concerns. The results on Uniref50 demonstrate that this method is stable enough to be suitable for larger vocabularies, which was my main methodological concern. I will increase my score.

We thank the reviewer for their positive feedback and for reconsidering their assessment. We sincerely appreciate your time and consideration.

Reviewer_SXq2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TC-NMD is a discrete diffusion framework that conditions denoising on trajectory history through a lightweight learned router. Independent re-corruption allows earlier token decisions to be revisited. The framework is instantiated on SLM and MDLM and evaluated on promoter and enhancer sequence design, with gains concentrated in the few-step sampling regime.

Critical strengths

The matched-backbone comparisons show consistent gains at low NFE across SLM and MDLM, making the few-step sampling benefit a meaningful contribution. The history and routing ablations help isolate the proposed mechanisms, and the motif-profile analysis provides supporting evidence independent of the FBD classifier.

Critical weaknesses
  1. The evaluation does not yet establish how reliably the metric improvements translate into better regulatory sequences. The datasets contain real genomic sequences, but generated sequences are assessed primarily through computational proxies. For enhancer generation, this concern is concrete: Fisher Flow Matching (Davis et al., NeurIPS 2024, Section 4.3 and Appendix F.5) reports test accuracies of 11.5% and 11.2% for the Dirichlet-FM classifiers on Melanoma and Fly Brain, and substantial changes in FBD when replacing the classifier with one trained from scratch. Since TC-NMD inherits this benchmark’s evaluators, their predictive quality and the stability of model rankings need to be established for the actual checkpoints used here. These earlier results do not establish the accuracy of TC-NMD’s exact checkpoints. Low classification accuracy alone also does not prove that the embeddings are uninformative, but the reported evaluator sensitivity makes independent validation important. The motif-profile results are useful, though they do not establish conditional biological activity.

For promoters, a small Sei-based MSE is difficult to interpret without target variance and a trivial reference such as a constant-mean predictor. Reporting normalized error would clarify whether the differences reflect meaningful conditional fidelity. The relationship between SP-MSE, the conditioning CAGE profile, and the Sei-derived activity target also needs a precise explanation; agreement with a surrogate activity score need not imply agreement with the requested position-specific transcription profile.

  1. The comparison does not sufficiently establish competitiveness against newer methods or distinguish sampling efficiency from final quality. In particular, ATGC-Gen (Su et al., TMLR 2025, Table 2) reports promoter MSE of 0.0192, versus 0.0267 for TC-SLM. These are independently reported results under potentially different conditions, so they are not a controlled head-to-head comparison. They do, however, motivate including this competitive published baseline under a consistent protocol. Within TC-NMD itself, Table 10 reports Fly Brain FBD of 0.57 for SLM+CFG at 100 NFE, compared with 2.02 for TC-SLM+CFG at 5 NFE in Table 1. This supports a distinction between low-step improvements and the best attainable quality. Extending the existing quality-versus-NFE plot and overhead analysis to stronger baselines would make the practical tradeoff clearer.

References:

Questions for authors
  1. Which exact FBD classifier checkpoints were used, what is their held-out predictive performance, and do the relative gains persist with an independently trained evaluator?

  2. What precisely are the prediction and target in SP-MSE? Can you report target variance, constant-mean baseline MSE, and a normalized measure of conditional fidelity?

8 comments
AuthorsOpenReview ↗

Response to reviewer SXq2 part 1

We thank the reviewer for acknowledging the contribution of our method to low-step sampling and recognizing our experiments and ablation studies. Below, we address the remaining concerns.

Q1,2, W1

W1a and Q1) Fair comparison with other baseline: We thank the reviewer for raising this concern. For a fair comparison, we used the released FBD classifiers from Dirichlet Flow Matching, with test top-1 accuracies of 11.23% and 11.45% on Fly Brain and Melanoma, respectively, reproducing the results reported by Davis et al. (2024). Their top-5 accuracies are 31.85% and 36.40%, and their macro AUROCs are 0.7379 and 0.7416, respectively.

To address the reviewer's concern, we trained one-stack and four-stack classifiers from scratch with three random seeds per architecture on each dataset, and our experiment shows that the improvement is not dependent on classifier choice. We evaluated the same generated sequences with each classifier. The table reports the ratio of mean FBD for TC-SLM to SLM under each evaluator and guidance setting (lower is better).

DatasetEvaluatorTest top-1Without CFGWith CFG
Fly BrainReleased classifier11.23%0.240.14
Fly BrainOne-stack (3 seeds)14.6–15.6%0.26–0.350.18–0.28
Fly BrainFour-stack (3 seeds)14.8–16.5%0.30–0.350.21–0.25
MelanomaReleased classifier11.45%0.570.48
MelanomaOne-stack (3 seeds)11.2–11.9%0.54–0.570.41–0.45
MelanomaFour-stack (3 seeds)11.4–11.9%0.56–0.600.47–0.53

The evaluators give broadly similar ratios within each dataset and guidance setting. TC-SLM achieved lower FBD across all evaluators, with and without CFG.

For the four-stack Fly Brain classifiers, selecting by validation accuracy alone picked three overfit checkpoints. Their test top-1 accuracies rose to 19.7–21.0%, but their real train–test FBD rose to 398–711. For these three runs, we selected the checkpoint with the highest validation top-1 among those with real train–validation FBD ≤ 5.0. We will include the full results in the revision. The additional experiments show that our improvements are not dependent on a single classifier, and our results are robust to evaluator choice.

AuthorsOpenReview ↗

Response to reviewer SXq2 part 2

W1b and Q2) SP-MSE scale: To address the reviewer's concern about the scale of SP-MSE, we calculated the variance of Sei-derived activity scores on real test promoters (0.0453) and used a constant-mean predictor as a reference. We also report normalized MSE (NMSE), obtained by dividing MSE by the target variance.

ReferenceMSENMSE
Constant-mean predictor0.04531.000
Shuffled pairing0.08931.970
SLM0.03030.669
TC-SLM0.02670.589

The constant-mean predictor which assigns the average Sei activity score to every promoter, and shuffled pairing which compares the Sei scores of randomly paired real promoters, are added as the baselines . Both SLM and TC-SLM perform substantially better than the simple baselines, with TC-SLM achieving lower normalized error, showing the effectiveness of our method.

To examine whether TC-SLM uses the conditioning information, we randomly shuffled the CAGE profiles across promoters and generated new sequences using these shuffled profiles. We then compared the Sei-predicted activity of the generated sequences with the original target activity scores. The MSE increased from 0.0269 to 0.0804 (2.99×), showing that changing the conditioning profiles significantly affects the generated sequences and their predicted activity.

We further examined the relationship between CAGE signal strength and Sei-predicted activity. On real test promoters, the Pearson correlation between log-transformed CAGE signal strength and Sei activity is 0.612. TC-SLM reproduces a similar correlation (0.616 ± 0.013), compared with 0.566 ± 0.013 for SLM. TC-SLM also better preserves the variance of real activity scores (0.79 versus 0.61 for SLM, relative to real data). These results further support that TC-SLM uses the conditioning information and better reproduces the relationship between CAGE signal strength and predicted regulatory activity observed in real promoters. We agree that these results do not establish position-specific transcriptional fidelity. To address this we additionally evaluated generated promoters using Puffin (Dudnyk et al., 2024), which predicts transcription initiation profiles. For each sequence we compute the Pearson correlation between Puffin's predicted initiation profile and the conditioning CAGE profile over the central 374 bp. Across all 7,497 test promoters, TC-SLM achieves a correlation of 0.405 ± 0.001 with the requested CAGE profiles, compared with 0.370 ± 0.001 for SLM and 0.504 for real DNA. To verify that the evaluation captures profile-specific agreement, we compared the predictions with CAGE profiles from other promoters. The correlations dropped to 0.155 for TC-SLM and 0.138 for SLM. These results show that TC-SLM better reproduces the requested positional transcription patterns.

Dudnyk et. al. Sequence basis of transcription initiation in the human genome.

W2a) Comparison with ATGC-Gen (Su et al., 2025)

We thank the reviewer for suggesting this comparison. We conducted additional experiments with ATGC-Gen and found that our method offers substantial advantages in sampling efficiency.

For promoter design, TC-SLM at 5 NFE achieves a lower MSE than ATGC-Gen-BERT at 1 NFE (0.0267 vs. 0.0334), while requiring less generation time (19.3 vs. 47.4 seconds per 8,192 sequences). Even with five sampling steps, our method is approximately 2.5× faster than their one-step generation. According to Su et al., ATGC-Gen-BERT achieves comparable promoter MSE using approximately 103 steps (0.0194), compared with 1,024 steps (0.0192). Under our H100 timing setup, the approximately 103-step configuration requires 4,666 seconds per 8,192 sequences, compared with 19.3 seconds for TC-SLM, corresponding to approximately 242× longer generation time.

For enhancer design, their BERT variant reports much higher FBD than TC-SLM (27.63 vs. 2.02 on Fly Brain and 45.27 vs. 4.82 on Melanoma), despite using a 23× larger model (85.5M vs. 3.67M parameters). Their strongest enhancer results are obtained using the autoregressive GPT variant, which achieves an FBD of 0.508 on Fly Brain. However, this model has 113.9M parameters (31× ours) and requires sequential generation of 500 nucleotides. In our H100 timing experiments, generating 8,192 sequences takes approximately 8,288 seconds with ATGC-Gen-GPT, compared with only 9 seconds for TC-SLM at 5 NFE. Therefore, while the GPT variant achieves approximately 4× lower FBD, it requires approximately 920× more generation time. We will make it clear in the final revision that our method is most useful in the low-sampling regime and include these additional comparisons and timing experiments in the paper.

AuthorsOpenReview ↗

Response to reviewer SXq2 part 3

W2b) Sampling efficiency and performance at higher NFEs

We thank the reviewer for raising this point. We agree that our method is most useful in the low-sampling regime, and we will make this clear in the revised paper and discuss its performance at higher NFEs.

At 5 NFE, TC-SLM substantially improves FBD on Fly Brain (14.65 → 2.02 conditional, 19.57 → 4.79 unconditional) and Melanoma (9.95 → 4.64 conditional, 10.11 → 5.74 unconditional). To further demonstrate its practical efficiency, we measured generation time on an H100 GPU (batch size 64, sequence length 500). On conditional Fly Brain, SLM requires approximately 17 NFE to approach the quality of TC-SLM at 5 NFE. Generating 8,192 sequences takes 9.0 seconds with TC-SLM versus an estimated 28.2 seconds with SLM, giving approximately 3.1× faster generation, including the computational overhead of our method. For one million sequences, this corresponds to approximately 18.4 versus 57.4 minutes, extrapolated from measured throughput.

We also investigated performance at higher NFEs and evaluated an adjusted version of TC-SLM on Melanoma. The following table reports FBD (conditional / unconditional), with all adjustments selected on the validation split and results averaged over three seeds.

NFETC-SLM (adjusted)Without re-maskingSLM
54.64 / 5.746.69 / 6.959.95 / 10.11
102.99 / 4.144.20 / 5.574.39 / 5.96
202.13 / 3.723.66 / 4.952.11 / 3.55
502.63 / 3.613.18 / 4.641.33 / 2.94
1002.66 / 3.593.48 / 4.391.21 / 2.82

Values are conditional / unconditional means; SDs are omitted for brevity. The adjusted method uses validation-selected re-masking budget (kk) and router temperature (τ\tau).

At higher NFEs, applying re-masking at every step can repeatedly reintroduce excluded candidates and prevent progressive refinement. We therefore limit re-masking to a fixed budget (k=16k=16 at 20–100 NFE), while also adjusting the router temperature where needed. The results show that re-masking remains beneficial compared with removing it entirely in higher NFE settings as well.

At very high NFEs, SLM still performs better. One possible reason is that the router must select useful information from a much larger number of history states (up to 99 at 100 NFE, compared with 4 at 5 NFE). More advanced router architectures may improve performance in this setting, which we leave for future work.

Overall, these results support our main motivation of improving generation quality with fewer sampling steps. This is particularly useful for large-scale biological sequence design, where many candidates are generated and screened before experimental testing.

For example, Bennett et al. (2023) generated approximately 2 million protein designs per target and selected around 20,000 for experimental testing, explicitly considering computational efficiency in their design pipeline.

Dai et al. (2026) further demonstrated that increasing the number of sampled protein candidates can improve the predicted quality of selected designs. These studies demonstrate the practical importance of efficient sampling in biological sequence design. We will include these additional experiments and clarify the low-sampling focus and higher-NFE limitations in the revised paper.

We hope that the additional experiments and analyses address the reviewer's concerns and further strengthen the empirical evidence supporting our method. We would appreciate it if the reviewer could reconsider their rating.

Bennett et al. Improving de novo protein binder design with deep learning.

Dai et al. Programmable design of functional proteins from natural language

Reviewer_sTWwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to improve discrete diffusion for DNA sequence design when earlier denoising decisions cannot be revisited and information from previous states is discarded. TC-NMD independently corrupts clean sequences at different noise levels, uses a learned router to compress previously visited states into a history representation, and introduces biased re-corruption to make intermediate decisions revisable. The main contribution is combining compressed trajectory conditioning with revisable sampling while retaining the original bidirectional denoiser. Experiments show consistent improvements over matched SLM and MDLM baselines at five sampling steps, with additional gains in promoter generation and motif-profile agreement.

Critical strengths
  • Learned trajectory conditioning: The model uses earlier denoising states without feeding the entire trajectory into the denoiser. The independent-corruption construction provides a valid probabilistic foundation for this approach, and the learned router offers a practical way to use trajectory information.

  • Consistent empirical gains: TC-NMD improves over matched SLM and MDLM baselines at five sampling steps, including conditional promoter generation. The motif-profile results provide evidence beyond FBD, while inference overhead remains modest.

Critical weaknesses
  • Population-consistency claim: Proposition 3 gives a valid pointwise bound, but the population-consistency claim in Appendix B is stronger. Minimizing expected cross-entropy need not minimize the biased KL under non-degenerate conditional label distributions. The statement should be narrowed or its assumptions clarified.

  • Inference specification: Section 2.3.3, the Appendix B introduction, and Algorithm 2 are inconsistent about hard-support projection. This is a clarification issue, not an established implementation error. The finite-KL result applies to the full-support reverse kernel.

Neither issue currently undermines the main empirical contribution or independently warrants rejection.

Questions for authors
  1. Is hard-support intersection actually applied during inference? Which sampling procedure produced the reported results?

  2. What assumptions are intended by the population-consistency statement in Appendix B? Is it restricted to realizable clean-label prediction?

  3. For conditional generation with classifier-free guidance, does the reported five-NFE budget count conditional and unconditional denoiser evaluations separately?

5 comments
AuthorsOpenReview ↗

Response to reviewer sTWw

We thank the reviewer for acknowledging the TC-NMD design and empirical gains. Below, we will address the reviewer’s concerns and questions.

Q1,W2:

We thank the reviewer for pointing this out. The “hard support sampling” that is mentioned in line 828 in Appendix B, is coming from the earlier drafts that we missed to remove. To be clear, in the current method and results, no hard sampling is applied, and the methodology section and the Algorithm are correct. We will remove that sentence from the paper. We need to mention this will not have any impact anywhere else in the paper.

Q2,W1:

We thank the reviewer for raising this. The proposition proved in Section B.3 does not show population consistency. The proposition proves that the per-step KL of our biased reverse kernel is bounded above by the cross-entropy, so minimizing the cross entropy minimizes an upper bound on the KL and brings it toward the bias-induced floor �0��0�. We will narrow the claim in (iii) in the appendix to “bounds the excess of the per-step KL above its bias-induced floor �1��1� using the cross-entropy”. However, we need to mention that this revision still supports our methodology, and our method does not rely on population consistency.

Q3:

The NFE for sample generation is counted as the number of denoising steps of the diffusion model. Therefore, in conditional generation, each denoising step consists of calling the denoiser model two times. We report five sampling steps, corresponding to five denoiser forward passes without CFG and ten denoiser forward passes with CFG (one conditional and one unconditional pass per step). Both TC-NMD and the corresponding baselines are compared under the same sampling-step convention. We will clarify this distinction in the revised manuscript.

Given that the reviewer noted that neither concern independently warrants rejection, we hope our clarifications have adequately addressed these issues, and we would appreciate it if the reviewer could reconsider their rating.

Dear Reviewer,

Thank you again for your time and feedback. We have addressed your concerns regarding the theoretical claims and sampling procedure in our rebuttal and revised the manuscript accordingly.

We would appreciate it if you could take a look at our responses and let us know if you have any remaining concerns or questions.

Thank you for your time and consideration.

Dear Reviewer,

Thank you again for your time and valuable feedback. We have addressed your concerns in our responses and would be grateful if you could review them before the rebuttal period ends. Please let us know if you have any remaining questions or concerns; we would be happy to provide further clarification.

Thank you for your consideration.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Updates to the revised manuscript

Dear Program Chairs, Area Chairs, and Reviewers,

We have uploaded a revised manuscript. Below we summarise the main updates and the changes to the MDLM results.

1. Updates to the manuscript

Few-step sampling and computational efficiency. We have clarified that our method is most useful in the few-step sampling regime. We now motivate the importance of efficient sampling in the introduction for biological sequence design where large numbers of candidates are generated and screened before experimental testing. We have also made this focus clear in the abstract and introduction, and added the limitation at higher NFEs to Appendix I.

We included additional experiments across 5 to 100 sampling steps in Appendix J.3, together with generation-time measurements. On conditional Fly Brain, TC-SLM at 5 steps achieves comparable FBD to SLM at approximately 17 steps, with an estimated 3.1× faster generation time, which also includes the computation overhead of our method.

Additional evaluation. To address the concerns about the FBD evaluator, we trained additional classifiers from scratch using different architectures and random seeds. TC-SLM achieves lower FBD across all evaluators, with and without classifier-free guidance (Appendix J.1).

For promoter design, we added the variance of Sei activity scores, constant-mean and shuffled-pairing references, and normalised MSE. We also conducted additional experiments to verify that the generated sequences respond to the supplied CAGE profiles. Using Puffin, TC-SLM achieves a higher correlation with the requested transcription profiles than SLM (0.405 versus 0.370). These results are included in Appendix J.2.

Theoretical clarification. We removed the outdated hard-support sampling statement from Appendix B, as no hard-support intersection is applied in our method. We also revised the claim in Appendix B.3 to clarify that cross-entropy bounds the excess of the per-step KL above its bias-induced floor. These changes do not affect the method or experimental results.

Computational cost and presentation. We added the computational measurements for the MDLM backbone in Table 6. The overhead is similar to SLM, with approximately 10% additional inference time and 1.4% additional peak inference memory. We also corrected the simplex notation in Equation 10 and clarified the figures and other presentation issues raised by the reviewers.

2. Updated MDLM results

While preparing the revision, we noticed that the guidance weights for the conditional MDLM experiments had been tuned using an earlier version of our code. We therefore repeated the tuning using the final implementation and updated the main results in Table 1 and the ablation results in Table 3. The sampling hyperparameters are reported in Table 5.

On conditional Fly Brain, the MDLM baseline improved from 9.79 to 6.15 FBD, while TC-MDLM improved from 6.07 to 2.46. With the updated results, TC-MDLM achieves 2.5× lower FBD than MDLM, compared with 1.6× in the original results. The absolute improvement also remains almost unchanged (3.69 versus 3.72).

On conditional Melanoma, the MDLM baseline improved from 14.34 to 7.44, while TC-MDLM remained at 6.48, corresponding to a 12.9% reduction in FBD.

All other results were already obtained using the best-tuned settings and remain unchanged. For unconditional MDLM, TC-MDLM achieves 1.7× and 1.6× lower FBD on Fly Brain and Melanoma, respectively. For TC-SLM, the improvements are 7.3× and 4.1× on conditional and unconditional Fly Brain, and 2.1× and 1.8× on conditional and unconditional Melanoma, respectively.

With these updated results, TC-NMD continues to outperform its corresponding Markovian baselines across both backbones, both datasets, and both conditional and unconditional generation.

AuthorsOpenReview ↗

Title: Summary of our response to Reviewer SXq2, and updates to the manuscript

Dear Area Chair and Reviewers,

We have posted a response to Reviewer SXq2, who asked for DNA results beyond Fly Brain and Melanoma and raised a concern about the quality of the classifiers used in the FBD evaluation. We addressed both points with additional experiments. We implemented our method on the DNA-Diffusion DHS dataset (DaSilva et al., Nature Genetics 2025). We added three types of metrics (k-mer JSD, MMD on k-mers, and metrics based on Nucleotide Transformer embeddings) and evaluated our results using these metrics, which do not depend on the FBD classifiers. We conducted additional ablations to show that the FBD metric is sensitive to changes in the DNA sequence distribution. We showed that our method improves the results on an additional DNA dataset, and that the improvements remain consistent across multiple metrics that do not depend on the FBD classifiers. We also showed that the FBD metric responds to changes in the DNA sequence distribution through controlled mutation experiments.

We uploaded a revised manuscript and added the material in Appendix Sections J.4 to J.6.

We thank you for your time and consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.