acceptodds
Withdrawn from ICLR 2027

CellRFT: Reinforcement Fine-Tuning for Single-Cell Perturbation Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Predicting cellular responses to perturbations supports the study of gene function, disease mechanisms, and therapeutic strategies. Despite advances in single-cell perturbation modeling, existing models typically optimize surrogate losses that do not directly reflect the biological criteria used for evaluation, so better data fitting need not yield better biological predictions. To address this mismatch, we introduce , a reinforcement fine-tuning framework that uses biological evaluation as direct training feedback. CellRFT uses policy-gradient optimization to learn from non-differentiable evaluations of generated cell populations and integrates multiple biological rewards through hierarchical reward aggregation. Comprehensive experiments demonstrate CellRFT's applicability across different pretrained models and effectiveness in improving perturbation prediction, reveal that optimizing one biological criterion can help or hinder others, and show that complementary rewards can improve criteria beyond those directly optimized, offering a way to probe how biological metrics shape model behavior, with the potential to inform evaluation design. Code will be made available.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Single-cell perturbation modeling, Virtual cells, Reinforcement learning, Diffusion models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_gfQNOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose CellRFT, a reinforcement fine-tuning framework for single-cell perturbation prediction. The goal is to address the mismatch between conventional training losses and biological evaluation metrics by using the latter as rewards in a policy-gradient optimisation procedure. The framework generates multiple candidate cell populations for each perturbation and assigns a shared reward to their constituent generation trajectories. To combine different biological objectives, the authors propose a hierarchical reward aggregation scheme that averages metrics within evaluation categories and subsequently across categories. The method is evaluated on the Replogle dataset using two variants of PerturbDiff. The authors report improvements in several differential-expression and perturbation-discrimination metrics and investigate the interactions between different rewards through a series of ablation studies.

Critical strengths

The mismatch between training objectives and downstream evaluation criteria is an important issue in single-cell perturbation modelling. Exploring whether these criteria can directly guide model optimisation is definitely a worthwhile direction.

Conceptually, extending policy-gradient optimisation to rewards computed on entire cell populations can be a reasonable approach to handling evaluation criteria that cannot be meaningfully computed for individual cells.

I also appreciated the experiments investigating how optimising individual metrics affects the others. The observed trade-offs, particularly the recall-related reward-hacking example, are interesting and could inform the design of evaluation objectives for perturbation models.

Critical weaknesses

The paper is generally readable, but I found the presentation somewhat repetitive. The motivation and the three main components of the framework are reiterated several times, while some important technical aspects of the actual optimisation procedure receive relatively limited explanation.

  • One of the core motivations for the approach is the potential non-differentiability of biological evaluation metrics. While this is true for metrics involving statistical significance thresholds, rankings, and discrete gene sets, it is less clear to me why reinforcement learning is preferable to differentiable approximations of these objectives -- e.g. expression fidelity, differential-expression effect recovery, and perturbation discrimination can be addressed through differentiable regression, ranking, or contrastive objectives. Moreover, population-level distribution matching is generally handled using established losses such as MMD or energy distance. The authors do not compare CellRFT with continued supervised fine-tuning using these objectives.

  • The experimental evaluation is restricted to one dataset and one train/test partition. Moreover, the manuscript does not report variability across random seeds, confidence intervals, or statistical significance tests. This is particularly concerning for the ablation studies, where small differences are used to support conclusions about reward composition and aggregation. For one, the advantage of hierarchical aggregation over flat averaging in Figure 4 is not particularly large. Given the stochastic nature of the generative model and the policy optimisation procedure, I believe repeated experiments and uncertainty estimates are essential to assess the reliability of the findings.

  • The authors repeatedly claim applicability across different pretrained models, but CellRFT is only applied to PerturbDiff, using two configurations of the same architecture. Although STATE, CellFlow, and other methods appear in the comparison, they are not used to assess the proposed fine-tuning framework.

  • I am not entirely convinced that the comparisons provide sufficient evidence for the effectiveness of the proposed framework relative to existing approaches. Several baseline methods were developed and trained for different objectives, and their reported performance may depend considerably on the training protocol and the evaluation setting. The manuscript would benefit from clarifying whether all baselines were evaluated under comparable conditions and whether additional adaptation was considered. More importantly, the authors do not include a compute-matched continued-training baseline.

Questions for authors
  • Q1: Can the authors compare CellRFT against compute-matched supervised fine-tuning using differentiable biological objectives?

  • Q2: Can the authors report results across multiple seeds (preferably different splits as well), with confidence intervals or standard deviations?

  • Q3: Can the authors evaluate CellRFT on additional datasets and generative backbones to support their claims of general applicability?

  • Q4: Can the authors clarify whether the competing methods were trained and evaluated under comparable conditions, and whether any task-specific adaptation or fine-tuning was performed?

I find the approach interesting, and I hope the authors can address these concerns and provide further evidence supporting their conclusions.

Reviewer_XJqSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CellRFT, a reinforcement fine-tuning framework that adapts Flow-GRPO to single-cell perturbation.

Critical strengths

The paper is well-written and easy to read

Critical weaknesses
  1. Conflation of training rewards and evaluation metrics. Of the 14 evaluation metrics, 11 are directly optimized during training. Only MSE, MAE, and ES are held out, and MSE actually worsens for the fine-tuned variant (0.0146 → 0.0149). This makes it difficult to assess whether the model genuinely predicts perturbation responses better or has simply learned to overfit to the optimized metrics. The authors should provide evaluations independent of the training rewards, such as distributional metrics (e.g., MMD, E-distance) or downstream biological validation.

  2. The claim of applicability "across different pretrained models" is overstated. The method is only evaluated on two variants of PerturbDiff from the same paper, on a single dataset (Replogle) and a single target cell line (HepG2). Architecturally distinct models such as STATE and CellFlow are not tested, and chemical perturbations as well as other datasets are not explored.

  3. Limited advantage over strong baselines. STATE outperforms CellRFT on DirAgr, LFCSpear, ES, MSE, MAE, PDCorr, and PDS_L1/L2. The simple per-cell-type mean baseline and the Linear baseline also achieve lower MSE. The paper mainly highlights the four metrics where CellRFT excels (DEOver, DEPrec, AUROC, AUPRC). Moreover, applying CellRFT to the fine-tuned variant degrades all three PDS metrics and MSE, which contradicts the claim of overall improvement.

  4. Limited methodological novelty. The core approach is largely a direct adaptation of Flow-GRPO, and lacks a sensitivity analysis for key hyperparameters such as GG and BB. The concurrent works D2R2 and PerturbCellRL are discussed only in passing, without any empirical comparison.

  5. Potential degradation of distribution modeling, with no diversity constraint or diagnostics. The objective (Eq. 7) relies solely on clipping and lacks KL regularization toward the reference model. Meanwhile, most rewards are based on pseudobulk means or DE statistics. Since the core strength of PerturbDiff lies in modeling response distributions, RL fine-tuning may induce mode collapse or a loss of cellular heterogeneity. Yet, the authors provide no diagnostic evaluation of this.

  6. No code is provided.

others: Figure 1(a) appears to be AI-generated,

Questions for authors
  1. Could the authors provide evaluations independent of the training rewards, such as distributional metrics (e.g., MMD, E-distance) or downstream biological validation (e.g., pathway enrichment, recovery of known regulatory relationships)?

  2. Could CellRFT be applied to architecturally distinct models such as STATE or CellFlow, and validated on datasets beyond Replogle or in chemical perturbation settings?

  3. With CellRFT, the model still underperforms STATE on most metrics, and the Finetuned variant degrades on all three PDS metrics and MSE. How do the authors account for this? Does it suggest that the benefit of the method is limited, or even negative, when starting from a stronger initialization?

  4. Beyond treating a population as a single sample and applying hierarchical averaging, what is the core technical contribution relative to Flow-GRPO? Could the authors provide a sensitivity analysis for GG and BB, as well as an empirical comparison with D2R2 and PerturbCellRL?

  5. How do the variance, sparsity, and cell-to-cell diversity of generated cells change before and after RL fine-tuning? Why is KL regularization toward the reference model omitted, and how would the results change if it were incorporated?

Reviewer_pgYSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

CellRFT fine-tunes a single-cell perturbation generator using population-level biological evaluation rewards. For each condition, it samples several cell populations, averages rewards within and across evaluation categories, and applies a clipped policy update with a population advantage shared across constituent cell trajectories. Experiments use one Replogle transfer protocol and two PerturbDiff initializations. The Scratch variant improves all 13 metrics reported in Table 1; the Finetuned variant improves nine. Single-reward and reward-combination experiments examine complementary gains and conflicting objectives.

Critical strengths
  • S1. Directly optimizing population-level biological evaluation is a reasonable way to address the mismatch between a generative training objective and the criteria used to assess perturbation predictions. Table 1 shows consistent improvements from the scratch checkpoint, including DEOver from 0.190 to 0.236, LFCSpear from 0.342 to 0.452, and AUPRC from 0.210 to 0.270. Improvements also extend to metrics excluded from the reward, such as ES, which increases from 0.623 to 0.771. These results make the approach worth investigating.

  • S2. The comparison of individual rewards, category combinations, and flat versus hierarchical aggregation is more informative than reporting only the final model. The experiments show that improving a target metric can hurt other criteria, and that some rewards support broader improvements.

    Appendix C provides a concrete example: a four-reward subset increases significant-DE recall from 0.382 to 0.833 while precision and PDCorr fall below the starting model. The authors acknowledge this failure mode and the PDS regressions of the finetuned initialization. This makes the study useful for understanding the consequences of choosing particular biological rewards.

Critical weaknesses
  • W1. Limited methodological novelty, with a more promising empirical contribution

    CellRFT is largely Flow-GRPO-style fine-tuning with population-level reward engineering: Equations (6)–(8) inherit group normalization and clipped transition-ratio updates. Under independent cell trajectories, shared population feedback follows standard score-function factorization. Equation (5) is fixed scalarization, with per-metric weights 1/3, 1/9, and 1/21 in the main configuration.

    The reward-interaction analysis in Appendix C offers a stronger source of originality: its refined three-reward subset improves 10 of 11 metrics. However, this remains an exploratory, retrospectively refined result; it does not yet establish that the analysis reliably selects complementary objectives. Selecting a subset using validation responses, freezing it, and testing it against simple same-size alternatives on held-out conditions would help establish a reusable reward-selection principle. This could strengthen the empirical novelty without requiring a new RL optimizer.

  • W2. Reproducibility is a major concern

    Essential details for evaluating and reproducing the RL update are missing: the PerturbDiff rollout sampler and stochastic transition density used in Equation (8), the justification for per-cell likelihood factorization, the denoising-step count, clipping threshold, training duration, and update/checkpoint schedule. The PDS training bank is also undefined, including its separation from validation/test response profiles.

Questions for authors

Questions For Authors

  • For the HITS-guided subset selection in Appendix C, which data supplied the single-reward response profiles and informed removal of DESigRec? Was the final subset fixed before held-out evaluation?
  • Please specify the sampler and transition density used in Equation (8), justify the per-cell likelihood factorization, and provide the complete RL configuration: denoising steps, clipping threshold, training duration, update frequency, and checkpoint selection. How are control/reference populations and the PDS reference bank constructed, with training, validation, and test responses kept separate?
  • Can you report seed-to-seed variability for the rewarded and excluded metrics, the hierarchical-versus-flat comparison, and the recall/precision/PDS trade-offs highlighted in the paper?
  • Do Table 1 and Figures 3–4 use different runs, checkpoints, or evaluation settings? LFCSpear improves by 0.110 in Table 1 but by 0.132 in Figure 3. Please reconcile these results and include DESigRec alongside the 13 main-table metrics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.