acceptodds
Withdrawn from ICLR 2027

MORAD: Reinforcement Learning with Multi-Objective Rewards for RNA Inverse Design

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce MORAD, an online reinforcement learning framework for backbone-conditioned RNA diffusion models that trains a single shared policy across diverse target backbones. Structure-conditioned RNA inverse design seeks nucleotide sequences compatible with a target three-dimensional backbone, yet existing reinforcement learning approaches either optimize a separate policy for each target backbone or rely on offline preference optimization, leaving online post-training of a reusable backbone-conditioned model largely unexplored. MORAD combines backbone-conditioned group sampling, group-relative reward normalization, and a forward-process diffusion objective. Beyond existing RL rewards that primarily target tertiary structural quality, MORAD jointly optimizes six measurements spanning tertiary geometry, base pairing, ensemble behavior, and sequence composition. To aggregate these heterogeneous objectives despite differences in scale and optimization direction, MORAD maps each measurement to a bounded desirability score and combines them with a weighted geometric mean. Compared with a tertiary-structure-only reward, this multi-objective formulation improves 10 of the 11 evaluation metrics that have a preferred direction. Applied to pretrained RIDE, MORAD improves GDT-TS by 16.0% (from 0.3300 to 0.3827) and pairing MCC by 19.4% (from 0.6110 to 0.7295), while reducing normalized ensemble defect by 27.5% (from 0.3906 to 0.2830). On the test set, MORAD outperforms RiboDiffusion, gRNAde, and RDesign across tertiary structural accuracy, secondary-structure pairing, and thermodynamic quality, establishing a new state of the art in structure-conditioned RNA inverse design.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
RNA inverse design, diffusion models, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5dgwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces an online reinforcement learning method for RNA inverse folding. The method, MORAD, applied RhoFold+'s structure metrics as reward function, and then applying DiffusionNFT as forward policy update to refine the model. The authors proved that the method can make RhoFold+'s structure scores generalized to other metrics like secondary structure prediction (EternaFold).

Critical strengths
  1. Comprehensive benchmark setting with reasonable metrics, and non-RL baselines.
Critical weaknesses
  1. The presentation assisted by LLM is pretty bad. Motivation, challenges, and description to the methods are all bad.
  2. The figures cannot convey the highlight of the methods. The AI-generated figures are ambigous and unclear.
  3. No RL-related baselines were involved in the benchmark given there were some RL-related works are cited.
  4. There is no novelty. This seems like a experimental report combining DiffusionNFT and an already existing reward function design.
Questions for authors

See weakness.

Reviewer_w2jUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces MORAD, an online reinforcement learning framework that post-trains a single shared backbone-conditioned RNA diffusion model (initialized from the 10.2M-parameter pretrained RIDE model) across multiple 3D target backbones:

  1. Shared-policy online RL: For each target backbone, an EMA policy samples G=8G=8 candidate sequences. Within-group reward standardization converts scalar rewards into weights for a DiffusionNFT-style positive/negative noise-prediction loss with a frozen-reference penalty
  2. Six-component reward: The reward combines three 3D alignment metrics scored by RhoFold+ (GDT-TS, TM-score, RMSD), two 2D/ensemble metrics scored by ViennaRNA RNAfold against the native sequence's RNAfold structure (pairing MCC and normalized ensemble defect), and nucleotide composition deviation, mapped via raised-cosine desirability functions and aggregated with a weighted geometric mean
  3. Validation: Trained for 390 updates on 527 RNA targets and evaluated on 153 test targets (71 from RNA3DB and 82 from RCSB), MORAD improves candidate-0 GDT-TS from 0.3300 to 0.3827, reported pairing MCC mean (via EternaFold) from 0.6110 to 0.7295, and normalized ensemble defect from 0.3906 to 0.2830 over pretrained RIDE
Critical strengths
  1. Practical shared-policy online RL for 3D RNA design. Unlike prior RL-guided diffusion for RNA 3D inverse design (such as RIDER), which optimizes a separate policy per target backbone, MORAD amortizes online feedback into a single reusable checkpoint using within-target group standardization and a forward-process denoising loss. This avoids per-target optimization at test time and enables direct inference on new backbones

  2. Cross-tool secondary-structure gains. While the training reward uses RNAfold predictions on the native sequence as the 2D reference, Table 2 evaluates EternaFold predictions against base pairs extracted directly from the native 3D structures. The reported pairing MCC mean improves by 19.4% (0.6110 to 0.7295; 133 valid targets for RIDE and 136 for MORAD), and mean F1 and Recall improve over the same 137 valid targets, reducing the concern that the policy simply overfits to RNAfold

Critical weaknesses
  1. Confounded reward ablation and unclear cross-run comparability (Sections 4.2–4.3, Table 4).

    • Confounded reward comparison: Section 4.3 (Table 4) compares "+ MORAD reward" against "+ Structural reward" (RctrlR_{\text{ctrl}}) to validate the multi-objective reward design. However, as shown in Table 5 (Page 13) and Appendix B (Page 14), switching from RctrlR_{\text{ctrl}} to the MORAD reward changes multiple factors at once: it adds the three auxiliary terms, replaces an arithmetic weighted sum with a weighted geometric mean (with a 10−310^{-3} floor), and alters the 3D desirability anchors and weights (e.g., GDT-TS changes from [0.15,0.70][0.15, 0.70] with weight 0.450.45 to [0.20,0.75][0.20, 0.75] with nominal weight 0.250.25, or 0.25/0.85≈0.2940.25/0.85 \approx 0.294 within-tertiary share). Because these changes are bundled together, Table 4 cannot isolate the contribution of the auxiliary terms (lines 411–412) or establish whether geometric aggregation itself improves performance over arithmetic aggregation
    • Cross-run comparability: Table 4 explicitly reports results from a separate evaluation run. Its “+ MORAD reward” scores differ from the main MORAD results in Tables 1–3: GDT-TS is 0.3694 vs. 0.3827, MCC is 0.6978 vs. 0.7295, and NED is 0.3256 vs. 0.2830. The reported MFE also differs from −26.49 to −30.149 kcal/mol. Although these differences do not by themselves establish an inconsistency, the paper does not sufficiently specify the training/checkpoint and candidate-evaluation differences between these runs. Clarifying these details would help assess the comparability of the reported results and the interpretation of the reward ablation
  2. Limitations in independent 3D evaluation, generalization assessment, and uncertainty reporting (Sections 3.3, 4.1–4.2).

    • Same-predictor 3D closed loop and point-estimate evaluation: Unlike the 2D evaluation (RNAfold for training, EternaFold for testing), target screening (native GDT-TS ≥0.30\ge 0.30), RL reward scoring (85% nominal weight), and final 3D evaluation all rely on the RhoFold+ predictor (Section 5, Appendix C–D). Without cross-evaluating on an independent 3D structure predictor, it is unclear how much of the 3D gain reflects predictor-independent structural compatibility versus predictor-specific optimization. In addition, Tables 1–2 use one candidate per target (candidate 0) and report no uncertainty estimates
    • Test–train similarity and unclear development protocol: The 153 test targets combine 71 targets from the RNA3DB test split with 82 curated RCSB entries. Across the evaluation set, the median maximum sequence identity to the 527 training targets is 0.6207, including 26 test targets (17%) with ≥0.80\ge 0.80 identity and 12 with ≥0.95\ge 0.95 identity (Section 3.3, Appendix C), and 3D structural similarity between the 82 RCSB targets and the training set is not reported. Because RL uses native training sequences as reward references, stratified results on the 71 RNA3DB targets and on low-similarity targets would help assess whether the gains extend beyond closely related targets. Moreover, Table 6 (Page 15) notes that there is no validation split while the 153-target evaluation set is evaluated 40 times during training; although the authors state these evaluations were not used for checkpoint selection, it remains unclear whether the reward parameters and training horizon (390 updates) were chosen independently of the evaluation set
Questions for authors
  1. What explains the reported differences between "+ MORAD reward" in Table 4 and "MORAD" in Tables 1–3 (including the difference in reported MFE between -26.49 and -30.149 kcal/mol), and do you have any controlled ablations that separate the 3 auxiliary terms from the switch to geometric aggregation and the modified 3D anchors/weights?

  2. From your existing evaluation records for RIDE and MORAD, do the gains hold when stratified into (a) the 71 RNA3DB test targets vs. the 82 RCSB targets, and (b) targets with maximum training sequence identity <0.60< 0.60 (or excluding the 26 targets with ≥0.80\ge 0.80)? Also, do you have a 3D structural similarity analysis for the 82 RCSB targets against the training set, and were the reward parameters and training horizon (390 updates) fixed independently of the 40 test-set evaluations?

  3. Do you have existing evaluations using a 3D RNA structure predictor other than RhoFold+, or uncertainty estimates under a consistent evaluation protocol for MORAD, pretrained RIDE, and the “+ Structural reward” policy, for example from existing eight-candidate evaluations?

Reviewer_FvkbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adapted the forward-process RL method for diffusion models, DiffusionNFT, onto RNA inverse design domain. Unlike previous papers about post training for RNA inverse design, the authors considered 6 measurements for the reward signal. They demonstrated the efficacy of their method through the improvement in the metrics about tertiary, secondary, and compositions over the baseline generation models.

Critical strengths
  1. The paper is well-written. The organization of different sections is sound and easy to follow. Most figures are informative and intuitive. Most experimental designs are clearly stated.
  2. The paper designed a multi-objective reward by incorporating measurements from various perspectives, i.e., Tertiary, Secondary, and Composition, which the previous paper failed to consider. And the paper showed the best performance over all the baselines across different measurements.
Critical weaknesses
  1. Some experimental designs are questionable.
  • It seems like the paper mainly compared with some conditional generation/diffusion models like RIDE and RiboDiffusion. But the paper did not show the comparison against some RL or other post-training methods for RNA inverse design, especially RIDER, an RL-finetuned method which is based on the same backbone as the paper, RIDE. I understand that the reward signal in RIDE is different from those considered in this paper. But at least comparing the performance in tertiary structure alignment is plausible since RIDER is also considering the tertiary structure as their reward signal. Also, it would be good to compare with more RL/post-training methods than RIDER.
  • The original DiffusionNFT paper was mainly applied to the continuous diffusion model which is suitable for images. I am wondering why the authors did not adopt a discrete diffusion/flow model for RNA sequence generation (discrete variables) but choose continuous domain instead by naively using continuous one-hot sequences. It seems there could be better way to model RNA sequences.
  • Could the authors explain why RIDER can only have" one policy per target"? Does "target" here main different tertiary structures (maybe different RNA families?) or different measurement types (tertiary, secondary, composition)? If it is former, how would MORAD resolve the issue?
  1. Some minor issues in paper presentation.
  • In figure 1, Does AiA_i represent the advantage? It would be better if the arrows of "standardize, clip +-2" points to AiA_i first then goes to rir_i. The subfigure to explain why the weights are grouped is less informative.
  • It is better to use coherent symbols for the sequences. In Line 152, the sequence is labeled as yy, but in Line 190 the symbol for the one-hot version is x0x_0. It would be better if both were using xx or yy.
Questions for authors
  1. What does "a reusable RL framework" mean? Does it mean that the policy is applicable to different backbones?
  2. In Line 46-47, what is the difference between RIDE and RIDER? It is better to highlight the difference at the start of the paper. Is it true that RIDE is the pretrained diffusion model without any post-training and RIDER is a baseline RL method fine-tuned on RIDE?
  3. How did you choose the weight values for different measurements in Line 262? What is the rationale here?
  4. Why don't MFE and EFE have a preferred direction in Table 3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.