acceptodds
Under review as a conference paper at ICLR 2027

Conditional Generation And Inpainting Of Non-coding RNA Sequences With Masked Discrete Diffusion

Abstract

Designing functional non-coding RNA (ncRNA) is fundamental to synthetic biology and RNA therapeutics, yet generative modelling for ncRNA has received far less attention than protein design. We present RNA-MDLM, a framework that extends Masked Discrete Language Models (MDLM) to the conditional generation and inpainting of ncRNA. We make two additions: first, conditioning on RNA-type representations from a pretrained RNA language model, and second, a modified classifier-free guidance scheme (Mod-CFG) that interpolates among conditional, unconditional, and random-sequence probabilities for better control. We also introduce REPAINT GAMES, a benchmark of seven structured masking tasks to probe a model’s performance on sequence patterns, structural motifs, and base-pairing per RNA type. Our model is trained on 4.6 million ncRNA sequences spanning six evaluable classes. It produces sequences whose composition and folding statistics closely match natural RNAs. Through extensive ablation studies, we find that the embedding-conditioned model achieves the best balance of structural fidelity, biological novelty, and inpainting accuracy, and its class label steers generation far more strongly than a plain label baseline. We further show that a model trained on a smaller, class-balanced subset can appear more realistic mainly by copying abundant natural sequences rather than learning their rules. We also benchmark against a masked-diffusion model and a family-specific VAE on ribozyme families, and find that a type-conditioned model like ours and a per-family model are solving different tasks, which must be accounted for in a fair comparison. We will release the code and the trained models.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Generative Models, RNA Sequence Design, Masked Discrete Diffusion, Inpainting, Conditional Generation, Non-Coding RNA

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_irahOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses the problem of conditional RNA sequence generation, an important and active area of research that remains computationally challenging. The authors aim to generate (or inpaint) class-conditioned RNA sequences with high sequence and structural fidelity. To achieve this, they propose RNA-MDLM, a Masked Discrete Language Model integrated with a modified variant of Classifier-Free Guidance (Mod-CFG). Additionally, the authors introduce RePaint Games, a benchmark designed to evaluate a model's ability to generate sequence patterns, structural motifs, and base-pairing interactions across different RNA classes.

Critical strengths
  1. The targeted problem is of high practical importance, and novel methodological approaches are welcome.
  2. The proposed RePaint Games benchmark is well-designed and provides strong utility for the broader research community.
  3. The authors invest significant effort into data preprocessing and curation, which is particularly crucial given the relative scarcity of established protocols for RNA Language Models.
  4. The exploration of the trade-off between sequence novelty and structural fidelity highlights a fundamental challenge in generative bio-modeling.
Critical weaknesses
  1. The ablation studies do not clearly isolate or demonstrate the individual contributions of all specific data curation choices and training objectives.
  2. The claim that adding random sequences to the training set improves performance is counterintuitive and lacks sufficient empirical support or mechanistic justification.
  3. Sequence novelty is evaluated relative to the validation set rather than the training set, which does not accurately measure memorization or true generative novelty.
  4. Key experimental results discussed in the main text are offloaded to the Appendix, which violates standard page-budget guidelines.
  5. The manuscript omits the exact number of training steps and total tokens/sequences seen by the model across reported checkpoints.
Questions for authors

Could the authors address the weaknesses 1, 2, 3, and 5?

1 comment

Q1. The ablations do not isolate the contributions of individual data curationand training choices.

Each ablation carries a clear message, and the numbers are consistent across all three levels of analysis. Removing redundancy and shrinking the data hurts. Cluster 80 and Cluster 90 against Vanilla:

metric (miRNA)VanillaClus80Clus90
2-mer JS improvement over random71.0%30.4%4.2%
MFE/nt (validation −0.439)−0.353−0.246−0.231
bp fraction (validation 0.722)0.6440.5750.565
G1 recovery (all types)0.6130.4420.405

Cluster 90 is close to random on 2-mer statistics for miRNA, and both cluster models fold into much weaker structures than training.

Class balancing alone did not help. The 50K model trails Vanilla on recovery in every game (G1 0.516 against 0.613, G3 0.490 against 0.597, G9 0.575 against 0.696) and on k-mer fidelity for most types (miRNA 40.7% against 71.0%, ribozyme 7.5% against 43.2%).

The 500K model did improve things, with caveats (see Q2).

Q2. The claim that random sequences improve performance is counterintuitive and under-supported.

The 500K model does recover statistics very effectively on some types. It is the closest of all six models to validation rRNA structure (MFE/nt −0.418 against −0.413; bp fraction 0.644 against 0.640), and it beats Vanilla on k-mer fidelity for tRNA (64.3% against 62.7%) and snRNA (53.9% against 44.4%).

Though the distributions hint towards memorisation, on rRNA, the 500K model is worse than random on 2-mer JS divergence (improvement over random −83.8%), its GC content is 0.623 against validation 0.526, and its normalised novelty drops to 0.053 (the lowest of any model-type pair) against 0.209 for Vanilla on the same type. It also over-structures tRNA and snoRNA (MFE/nt −0.391 against −0.365, and −0.279 against −0.220).

Q3. Novelty is measured against the validation set, not the training set.

Accepted, and we have redone it against the training set. We see no concerning similarity; the generated sequences are further from the training data than real held-out RNA is, in 34 of 35 conditions by edit distance and 32 of 35 by alignment identity, with an exact match rate of 0.0 everywhere.

variantLevenshtein gen / natvsearch gen / nat
Vanilla0.656–0.889 / 0.884–0.9780.413–0.975 / 0.880–0.993
Emb. Cond.0.666–0.908 / 0.884–0.9780.448–0.991 / 0.880–0.993
Subset 500K0.749–0.954 / 0.878–1.0000.722–0.986 / 0.882–0.999
Cluster 800.583–0.732 / 0.678–0.7470.076–0.970 / 0.593–0.879
Cluster 900.556–0.751 / 0.773–0.8510.001–0.880 / 0.752–0.911

Q4. The exact number of training steps and tokens/sequences seen is missing.

As requested, we are mentioning the training steps here; the details of the datasets are provided in the original text

variantoptimizer steps
Vanilla (full data)352,917
Embedding Conditioned289,570
Cluster 8054,520
Cluster 9091,800
Subset 500K96,875
Reviewer_xKZzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RNA-MDLM combines masked discrete diffusion, RNA-type conditioning, and discrete RePaint for ncRNA generation and inpainting. A roughly 130M-parameter model uses learned labels or frozen NucleicBERT class prototypes. A balanced-subset variant uses two-reference Mod-CFG. Seven masking tasks and computational generation assessments evaluate recovery and sequence/structure statistics. The clearest comparative result is pooled recovery of 0.53 versus 0.47 for EvoFlow-RNA; the incremental value of prototype conditioning and modified guidance remains less clear.

Critical strengths

A unified generation/inpainting interface and seven structured masking tasks address useful RNA-editing capabilities. Table 2 uses the same masked sequences across models and reports improved pooled recovery over EvoFlow-RNA (prototype 0.53, learned label 0.52, EvoFlow 0.47), with paired statistics. This supports the system even though it does not isolate prototype benefits (pp.5,8-9).

The family evaluation discloses important limits: class-versus-family conditioning, weak strict Pistol membership, and close training neighbors for many reference members. These qualifications and the per-type recovery plots make the computational evidence more informative; no functional validation is implied (pp.7-9,30).

Critical weaknesses

The central prototype-steering claim rests on inconsistent reporting. Table 8 assigns prototype recovery 0.25 to the wrong label, while the actual G6 plot places approximately 0.25 under the unconditional label and visibly higher recovery under the wrong label. The G6/G7 captions are interchanged. Section 5.3 also claims G7 superiority for every class, contrary to Figure 24. Table 2 reports no statistically distinguishable pooled advantage over learned labels, and the training settings differ. These findings limit attribution to pretrained biological information; they do not invalidate the overall recovery gains (pp.6-7,9,24-25,30).

Mod-CFG is used only in Subset 500K, alongside changes to data composition and label dropout, so the added value of the random-sequence reference is not isolated. Equations 1-2 also lack a zero-mass fallback: at gamma=5, identical conditional/reference distributions yield s=-3p, whose positive part is zero everywhere. This is an underspecified mathematical boundary, not evidence that the implementation failed or reported runs encountered it (pp.4-6).

Questions for authors
  1. Which correct/null/wrong-label values should govern the G6 steering claim, and after reconciling Table 8 with the actual G6 panel, what conclusion about prototype conditioning remains supported, including the G7 comparison?

  2. What does the random-sequence reference add beyond the balanced unconditional reference in the existing evidence, and what sampling fallback defines Mod-CFG when all clamped weights are zero?

1 comment

Q1. Which correct/null/wrong values underlie Table 8 and the G6 panel? Reconcile the captions and the every-class G7 claim. Beyond a recovery drop, what shows that wrong-label conditioning yields the requested type rather than degraded completions?

The column headed “Wrong label” holds the null-label values. Correct per-type numbers:

Vanilla corr / null / wrongEmb. Cond. corr / null / wrong
miRNA0.586 / 0.567 / 0.4540.599 / 0.249 / 0.464
rRNA0.936 / 0.848 / 0.8680.935 / 0.249 / 0.907
ribozyme0.556 / 0.459 / 0.4140.545 / 0.253 / 0.429
snRNA0.637 / 0.616 / 0.5390.632 / 0.255 / 0.542
snoRNA0.565 / 0.514 / 0.4450.573 / 0.251 / 0.455
tRNA0.750 / 0.554 / 0.4010.784 / 0.248 / 0.487
pooled0.672 / 0.593 / 0.5200.678 / 0.251 / 0.547

The G6 and G7 captions are swapped. We corrected these issues. The claim that the prototype model outperforms Vanilla on every G7 class is overstated: it wins tRNA only, and pooled Vanilla is ahead.

gap (correct − null)miRNArRNAribozymesnRNAsnoRNAtRNA
Vanilla+0.020+0.088+0.097+0.021+0.051+0.197
Embedding Conditioned+0.350+0.685+0.292+0.377+0.322+0.537

Embedding conditioning wins on all six types. The mechanism is visible in the absolute numbers, with no label the prototype model falls to 0.248–0.255, so its generation is driven entirely by the label, whereas the learned-label model still recovers 0.51–0.85 from sequence context and therefore depends less on the label. That is what conditioning strength means here.

Q2. With matched budgets, do pretrained prototypes beat learned labels and random fixed prototypes? On Subset 500K, what is gained by the balanced reference and then by the random reference?

We will run the three matched-budget arms, learned label, NucleicBERT prototype, and a random fixed prototype through the same projection, at identical learning rate, batch size, and step count, and report whatever they show. Until then, we restrict the prototype claim to what Table 2 supports, which is indistinguishability. Further on the games, embedding conditioning wins five of seven (listed in the reply to gCov), which is a consistent margin.

Q3. Were prototypes computed only from training sequences? Report generated-to-training nearest-neighbour identity against a natural reference, and a cluster-disjoint subset. For G7, does the advantage hold under pairing consistency?

Prototypes were computed from the full annotated corpus, so validation rows were included. We recomputed them from the training split alone:

classcosinen (full)n (train only)
rRNA1.0000001,979,5241,583,216
tRNA1.0000002,445,3671,956,680
snoRNA1.000000136,027108,859
miRNA0.99999941,76633,530
ribozyme0.99999718,42514,617
snRNA0.9999957,1735,746
lncRNA0.999931697534

The leakage is real, but it is not consequential. Generated-to-training identity with a natural reference, both scored against the same training corpus; see irah Q3

Q4. Eqs. 1–2 are undefined when clamping leaves zero total mass; at γ=5 with three identical uniform distributions, every signed weight is −0.75. What fallback, and how often? Also: how does j enter Algorithm 1, are known positions preserved exactly, and what is the denoiser-call budget per model?

The clamp. Your arithmetic is right, and §3.2 does not say what follows from it. §3.2 is clear that s is not a probability; entries may be negative by design and Eq. (2) keeps the positive part, but it does not state what happens when no nucleotide is positive, which is the case you identified. At γ = 5 a nucleotide survives only when p_c[k] > 0.8·(p_∅[k] + p_rand[k]). Early in the reverse process, almost every position is masked, so all three predictions are near-uniform and no nucleotide clears that bar. In that case, Eq. (2) has nothing to renormalise.

How j enters Algorithm 1. The schedule is built in chunks of j steps: the reverse process runs to the end of a chunk, re-noises back j steps, and repeats R times before moving on. At j = 1 this is the inner loop as printed, so Algorithm 1 describes only that case.

Known positions. During the trajectory they are overwritten with ground truth noised to the current level, which is what lets information flow between known and generated regions; after the final denoising step the original tokens are restored exactly.

Denoiser-call budget. RNA-MDLM at T=250, R=10, j=10: about 2,500 calls per sample. EvoFlow-RNA: 128 ancestral steps. At matched calls (T=128, R=1, j=1, same sequences and masks) EvoFlow reaches 0.473 and RNA-MDLM 0.414, though at R=1, j=1 RePaint’s resampling is off, while 128 steps is EvoFlow’s native setting.

Reviewer_gCovOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RNA-MDLM for conditional generation and inpainting of non-coding RNA, using RNA-type conditioning and a modified classifier-free guidance scheme. It evaluates generation statistics and structured inpainting tasks through the REPAINT GAMES benchmark.

Critical strengths

None

Critical weaknesses
  1. The evaluation relies too heavily on mean recovery rate. Recovering a reference sequence does not, by itself, establish that generated sequences have useful structural or functional properties. The relationship between recovery rate and downstream design quality is not sufficiently demonstrated. A validation-set ELBO, together with stronger downstream design metrics, would provide a more informative assessment. In particular, the reported folding statistics should be connected more directly to whether generated RNAs satisfy their intended design objectives.

  2. The technical advance over EvoFlow-RNA is not sufficiently clear. The paper needs to explain which contributions go beyond adapting existing approaches and provide evidence that these changes improve RNA design. The design evaluation also remains too focused on motif-oriented inpainting to establish broader utility for de novo or function-conditioned RNA design.

Overall, I recommend rejection because the evaluation does not convincingly establish downstream design quality, and the methodological advance remains unclear.

Questions for authors

What evidence links mean recovery rate to downstream structural or functional quality? What are the substantive methodological differences from EvoFlow-RNA, and which results isolate their benefits? How far do the existing generation experiments establish utility beyond motif-oriented inpainting?

1 comment

Q1. What evidence links mean recovery rate to downstream structural or functional quality?

The recovery rate shows that the model rebuilds the important parts of a sequence from limited context. However, the evaluation does not rely on recovery alone. We also report sequence-level and structure-level metrics for all six types.

Sequence level, 1000 samples per type, Embedding Conditioned against the held-out validation split (Table 6):

typeGC gen / val2-mer JS3-mer JS4-mer JSidentity (%)
tRNA0.543 / 0.5670.0350.0520.07576.2
rRNA0.513 / 0.5270.0250.0490.09580.4
miRNA0.459 / 0.4510.0250.0440.07370.7
ribozyme0.501 / 0.5460.0470.0780.12666.3
snRNA0.432 / 0.4450.0460.0760.12075.1
snoRNA0.406 / 0.4460.0600.0800.10161.0

Structure level, folded with ViennaRNA against length-matched validation

sequences (Table 4):

typeMFE/nt gen / valbp fraction gen / val
tRNA−0.337 / −0.3690.601 / 0.616
rRNA−0.357 / −0.4130.641 / 0.640
miRNA−0.379 / −0.4390.675 / 0.722
ribozyme−0.327 / −0.4060.607 / 0.644
snRNA−0.221 / −0.2460.541 / 0.558
snoRNA−0.213 / −0.2150.549 / 0.544

The generated sequences closely match the properties of validation sequences. We have added an analysis to compare novelty against the training corpus, with two metrics and a natural-sequence control. Generated sequences are further from the training data than real held-out RNA is, in 34 of 35 conditions by edit distance and 32 of 35 by alignment identity. Exact match rate is 0.0 in every single condition. So recovery is not achieved by memorization.

Q2. What are the substantive differences from EvoFlow-RNA, and which results isolate their benefits?

  1. Our model does conditional generation. The public EvoFlow-RNA (mini, 33M)checkpoint carries no RNA-type label, so the cross-type game cannot run on it at all.
  2. Ours is trained from scratch as a diffusion model rather than built on a pre-trained encoder such as RiNALMo.
  3. RePaint Games: seven structured inpainting tasks with clear success criteria.
  4. A standardised annotated corpus: 4.6M sequences with RNA type, Rfam family, and GO terms.
  5. Embedding conditioning and the Mod-CFG guidance scheme.

The games isolate (1) and (3). Head-to-head on the six games EvoFlow can run

(mean recovery over all types and variants):

gameRNA-MDLM (EmbCond)EvoFlow-RNA
G1 random masking0.6230.522
G2 contiguous block0.5500.483
G3 motif recovery0.6070.485
G4 flanking context0.3940.397
G5 interior context0.4470.448
G7 structure preservation0.6900.530
G6 cross-type0.492not possible

We win four of six clearly, and the two we lose (G4, G5) are losses by 0.003 and 0.001. The largest gaps are motif recovery and structure preservation, which are the tasks the method was designed for.

The inference budgets are as follows: RNA-MDLM at T=250, R=10, j=10 uses about 2,500 denoiser calls per sample; EvoFlow uses 128, at 130M against 33M parameters. At matched calls (T=128, R=1, j=1, same sequences and masks), EvoFlow reaches 0.473 and RNA-MDLM 0.414, but at R=1, j=1 the resampling that constitutes RePaint is switched off, while 128 steps is EvoFlow’s native setting. Table 2 in the revision carries a denoiser-call column and both rows.

On embedding conditioning: it beats the learned label on five of seven games (G1 0.623/0.613, G2 0.550/0.535, G3 0.607/0.597, G4 0.394/0.382, G5 0.447/0.440) and loses on two (G6 0.492/0.595, G7 0.690/0.696). Table 2 does not separate it from learned labels, as xKZz says, and the matched-budget controls are in the revision.

Q3. How far do the experiments establish utility beyond motif-oriented inpainting?

Sequence-level and structure-level analysis together show the model generates sequences appropriate to the requested class without copying training data. GC, k-mer, and MFE statistics align with the per-type validation distributions (tables above). Games 3 and 9 mask by structure, loops and stems, and the 3′ side of base-paired stems, rather than at random, and G9 is where our margin over EvoFlow is largest (0.690 against 0.530).

We accept that this is still narrower than de novo or function-conditioned design. A cross-model de novo comparison is not a route we can take because the available baselines do not support length control, so their generations cannot be matched to ours length-for-length, and every composition and structure metric we report is length-sensitive.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.