acceptodds
Under review as a conference paper at ICLR 2027

Beyond Reconstruction: Likelihood-Free Reinforcement Learning for Virtual Cells

Abstract

Virtual cell models predict how cells respond to genetic perturbations. A central use of these predictions is deciding which perturbations can shift disease-relevant pathways in a desired direction and accelerate therapeutic target discovery. Because these models are trained to reconstruct gene-level expression, they can reduce reconstruction error without ranking the perturbations with the strongest pathway effects near the top. Pathway-level effects are defined over gene sets, relative to a background set, rather than individual genes so they are difficult to express as supervised losses but can be directly scored as rewards, which suggests benefits from reinforcement learning (RL). Standard RL, however, requires policy likelihoods, whereas virtual cell models generate high-dimensional cell populations without tractable likelihoods. We introduce Maximum Mean Discrepancy Policy Optimization (MMD-PO), a likelihood-free reinforcement learning method that weights rollouts sampled from a frozen policy by biological rewards, and moves the current policy toward this reward-weighted empirical target by minimizing maximum mean discrepancy between rollouts. We pair MMD-PO with Gene-Effect and Pathway-Enrichment rewards. In a controlled synthetic benchmark with known latent effects, MMD-PO improves held-out ranking over continued supervised training, GRPO, and online DPO, and its gains depend on feedback reliability, candidate diversity, and the headroom of the starting model. On held-out perturbations in HepG2 cells and zero-shot transfer from resting to stimulated primary T-Cells, MMD-PO improves top-% hit enrichment over GRPO by % and %, respectively, while retaining prediction fidelity. These results establish likelihood-free policy optimization as a practical interface between virtual cell models and experimental prioritization.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Reinforcement Learning, Single-Cell Perturbation Prediction, AI for Science, Virtual Cell

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_oxYtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Maximum Mean Discrepancy Policy Optimization (MMD-PO) for reward-guided post-training of virtual-cell models. The method reweights cell populations sampled from a frozen policy using biological rewards and a reference-distance penalty, then fits the current policy to the resulting empirical target through MMD. It combines gene-effect and pathway-enrichment rewards to improve perturbation prioritization. Evaluation includes retrospective experiments across three backbones, held-out HepG2 perturbation responses, transfer between resting and stimulated T cells, and controlled synthetic studies examining when post-training helps.

Critical strengths
  1. A useful post-training interface for population-generating models. MMD-PO avoids evaluating output likelihoods and differentiating through the reward, while operating directly on generated cell populations. This is a practically relevant construction, and evaluation across multiple backbones provides some evidence of portability.

  2. Promising prioritization results accompanied by boundary-condition analysis. Some improvements are substantial: for X-Cell on HepG2, EF@1% increases from 6.714 with GRPO to 10.300 with MMD-PO. Reporting both prioritization and predictive-fidelity metrics, together with controlled synthetic experiments, helps characterize the benefits and limitations of the approach.

Critical weaknesses
  1. The missing direct task-loss baseline limits attribution of the central contribution. Both rewards measure agreement between predictions and observed responses. An important alternative is therefore to augment the native supervised objective with the same gene-effect and pathway-effect errors. The existing pathway-gene SFT baseline restricts the genes covered by the native loss, while reward-weighted SFT changes example weights; neither directly tests this alternative. At minimum, the differentiable Gene-Effect Huber loss could be optimized directly. For the pathway component, a clearly specified differentiable implementation or surrogate could be evaluated with validation-based loss-weight tuning. The same-reward GRPO/Flow-GRPO comparisons already provide partial attribution evidence, but they do not establish whether MMD-PO offers an advantage over this straightforward task-aligned optimization route. This distinction materially affects the claimed algorithmic contribution.

  2. The additional benefit of the MMD update is not sufficiently isolated from the broader training recipe. The method combines candidate generation, reward weighting, reference-distance penalization, and MMD fitting. A comparison that holds the candidate populations and weights fixed while varying the fitting objective would help determine whether the benefit specifically comes from MMD or more generally from learning from reward-weighted generations. Moreover, the PerturbCellRL-inspired comparison omits an original reward component, and some compatibility experiments differ in rewards, optimization settings, or computational conditions. These limitations are disclosed, but the resulting comparisons do not cleanly establish superiority of the update mechanism over the closest likelihood-free alternatives. Stronger attribution is important for assessing the significance of the proposed algorithm.

Questions for authors
  1. Can you hold the reward-weighted candidate populations fixed and compare MMD with alternative distribution-matching or weighted-fitting objectives? A comparison against a close likelihood-free baseline using the same reward, initialization, and a comparable training budget would also clarify the update rule’s independent contribution.

  2. The practical algorithm replaces the reference-to-old-policy likelihood ratio with a reference-output distance penalty and replaces KL projection with MMD projection. The manuscript appropriately identifies the practical weights as a surrogate. Please clarify which properties of the original KL-regularized objective, if any, are claimed to carry over to the implemented update, and which derivations serve only as motivation.

7 comments

Thank you very much for your positive comments and appreciation of our novelty, experiments and presentation. Below are our responses to the specific query raised.

Weakness 1: Can the gains be obtained by directly adding the biological task losses to the native supervised objective?

Thank you for your thoughtful comment. As requested, we added the Gene-Effect Huber loss to the native supervised objective and swept its coefficient. Below, we report the setting with the highest EF@1% in the sweep, alongside the original supervised-objective and reward ablations. The results are updated in Table 2.

MethodEF@1%EF@5%AnDCG-Rank@1%AnDCG-Rank@5%AnDCG-Geo@1%AnDCG-Geo@5%
Continued SFT5.3714.9030.3400.3020.3970.396
Pathway-gene SFT3.9912.7640.2110.1570.3200.258
Reward-weighted SFT4.4633.3430.2550.1850.3300.277
SFT + Gene-Effect Huber loss3.1302.4200.1230.1100.1730.178
MMD-PO: Gene-Effect reward only9.2105.5500.4180.3360.4650.435
MMD-PO: Pathway-Enrichment reward only6.2005.1600.3940.3340.4470.445
MMD-PO: Combined reward10.3005.6100.4250.3390.4810.445

The direct Huber-loss baseline does not beat the performance of MMD-PO. These results support the benefit of reward-weighted population fitting over the tested direct-supervision formulation.

For the pathway component, GSEA enrichment depends on a hard ordering of genes. Standard backpropagation does not provide gradients through changes in these discrete rank assignments. Weighted GSEA can retain gradients through score magnitudes within a fixed ordering, but would not be the same calculation as the original enrichment reward. Therefore directly adding the pathway reward as a loss won’t be suitable for our cases.

Weakness 2 and Question 1: Can you hold the reward-weighted candidate populations fixed and compare MMD with alternative distribution-matching or weighted-fitting objectives?

Thank you for your insightful question. To address this concern, we added comparisons replacing the outer MMD with Wasserstein distance and Sinkhorn matching. With the same initialization and optimization budget, we changed the MMD with other distribution matching functions:

Outer objectiveEF@1%EF@5%AnDCG-Rank@1%AnDCG-Rank@5%AnDCG-Geo@1%AnDCG-Geo@5%
Wasserstein5.7794.4510.3690.2930.4540.420
Sinkhorn4.9704.3570.3310.2880.4330.421
MMD10.3005.6100.4250.3390.4810.445

MMD achieves the highest reported performance across all six metrics. These results support a contribution from the MMD fitting objective beyond candidate generation and reward weighting. We will include the transport costs, Sinkhorn regularization, and optimization details in the revision.

On the query in W2: Is the PerturbCellRL comparison fair given the omission of its original pathway reward?

PerturbCellRL uses DiffusionNFT as its likelihood-free update mechanism. We reconstructed its Pearson, RMSE, and DE-Spearman rewards from the June 2026 description, omitting the pathway verifier because its published annotations cover Norman and we lacked corresponding validated annotations for HepG2. We therefore label the baseline PerturbCellRL-inspired.

To address the reward mismatch, we additionally evaluated DiffusionNFT using our Gene-Effect and Pathway-Enrichment reward. We compare this new result with the original manuscript’s reconstructed-3R and MMD-PO results on HepG2 below.

MethodEF@1%EF@5%Rank AnDCG@1%Rank AnDCG@5%Geo AnDCG@1%Geo AnDCG@5%
DiffusionNFT + reconstructed 3R3.9002.7990.1430.0990.1840.162
DiffusionNFT + our reward1.5101.9200.0510.0620.1220.119
MMD-PO + our reward6.5653.0150.1650.1200.1780.169

MMD-PO achieves higher point estimates than DiffusionNFT with the same reward across all six endpoints, supporting an advantage over this closely related likelihood-free alternative in the evaluated setting. We will revise the manuscript to clarify the baseline construction and include this additional comparison using the same reward.

Reviewer_oxYtOpenReview ↗

Thanks for the Response

Thank you for the additional experiments. A few points remain:

  1. Inconsistent numbers. The Wasserstein/Sinkhorn results in this thread and in the 9MxJ thread have identical AnDCG values but different EF values (e.g., Wasserstein EF@1% 7.240 vs 5.78; Sinkhorn 6.150 vs 4.97). Please clarify which values are correct. I note that 8.93 ± 1.45 is the five-inference-seed mean; please state whether 10.30 and the new baselines are single-inference-seed results or averages over multiple inference seeds.

  2. Attribution to MMD. Mean L2 (reported to 9MxJ) recovers 70–89% of the improvement over SFT initialization across the six metrics. MMD has higher point estimates on all six, but without matched uncertainty estimates it is unclear whether its additional gains are robust. Could you report mean ± SD for all objectives using the same inference seeds and evaluation protocol, and confirm whether the candidate populations and target weights were held fixed across objectives?

  3. Direct task-loss baseline. The best Huber configuration (EF@1% 3.13) is well below continued SFT (5.37). Please report the coefficient range, a λ = 0 control, and how the configuration was selected. This experiment also covers only the Gene-Effect term; a differentiable pathway surrogate, as suggested in my review, has not been evaluated.

  4. DiffusionNFT. Which backbone and budget were used (MMD-PO EF@1% is 6.565 here)? When switching to your reward (EF@1% 3.900 → 1.510), were the reward scale and optimization hyperparameters re-tuned, and how?

I will reconsider my score after these clarifications.

Reviewer_9MxJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MMD-PO, a likelihood-free method for post-training virtual cell models toward biological objectives relevant to perturbation prioritization. The method generates candidate cell populations from a frozen policy, scores them using biological rewards, and constructs a weighted empirical target. An energy-distance penalty relative to a supervised reference provides a practical surrogate for the otherwise intractable reference-to-behavior likelihood-ratio correction. The current model is then trained to match this target through a sample-based MMD objective.

The construction operates at two distributional levels: energy distance, itself equivalent to a squared MMD under an appropriate cell-level kernel, compares individual cell populations; a Gaussian kernel built from this distance supports an outer MMD over distributions of populations. The biological rewards combine agreement in gene-level perturbation effects with agreement in pathway enrichment relative to background genes. Experiments across several virtual cell backbones, together with controlled synthetic studies, demonstrate improvements in perturbation prioritization over the tested supervised and reward-guided baselines.

Critical strengths

The main strengths are:

  1. An interesting and novel approach to virtual cell post-training. The paper addresses a meaningful gap between expression reconstruction and experimental prioritization. Its output-space formulation offers a useful way to incorporate biological feedback without requiring tractable population likelihoods or differentiable rewards.

  2. Intuitive and empirically effective surrogates. Favoring high-reward candidates while penalizing departures from a supervised reference is a natural construction. The biological rewards are interpretable, and the empirical comparisons support the usefulness of combining them with distribution matching.

Critical weaknesses

My main concerns are about the theoretical characterization and justification of the distributional objective:

  1. The distribution-over-distributions structure deserves a more precise treatment. Each output population can be represented as an empirical distribution of cell states, so the policy induces a distribution over such empirical distributions. The nested MMD construction therefore compares a richer object than a single distribution of cells. Although Appendix A.3 establishes positive semidefiniteness of the population kernel, the paper does not sufficiently explain what properties of the population-generating distribution the outer MMD identifies, what information the construction preserves, or how its interpretation depends on population size and within-population sampling. A clearer formalization would help distinguish matching pooled cell distributions from matching variability across population-level predictions. It would also clarify how the practical surrogate-weighted MMD projection relates to the original KL-regularized reward objective.

  2. The choice of nested MMD is insufficiently isolated from other suitable distributional objectives. The experiments demonstrate the effectiveness of the overall framework, but do not establish why this particular geometry is preferable. Comparisons with Wasserstein distance and Sinkhorn divergence, a Gaussian approximation using means and covariances with Bures–Wasserstein distance, and even a simple squared L2 distance between population means would be informative. This is particularly relevant because the biological rewards primarily depend on pseudobulk summaries. Such comparisons could reveal whether the gains require modeling population heterogeneity or largely arise from improving mean responses. The alternatives should be evaluated with candidate generation, rewards, target weights, and optimization budgets held fixed, while clearly distinguishing changes to the inner population comparison from changes to the outer matching objective.

Questions for authors
  1. Can you formalize the policy as a distribution over empirical cell distributions and characterize what the outer MMD identifies? In particular, is the proposed population kernel characteristic on the relevant space, and how does population size affect the interpretation?
  2. What relationship can be established between the practical surrogate-weighted MMD update and the original KL-regularized reward objective? Which connections are exact, and which are heuristic?
  3. How does the method compare with the alternative distributional objectives above under a matched training protocol? Do the ranking improvements require information beyond population means or first and second moments?
3 comments

We thank the reviewer for the constructive feedback and for recognizing the novelty, theoretical motivation and performance of our paper. We address the reviewer’s concerns in three parts: (1) the interpretation and identifiability of the population-level objective (W1, Q1), (2) its relationship to the KL-regularized reward objective (W1, Q2), and (3) matched comparisons with alternative objectives and the role of population heterogeneity (W2, Q3).

[W1/Q1]: What does the outer MMD identify about the distribution of generated cell populations, and how does this depend on kernel characteristicness and population size?

Thank you for highlighting this distinction. For a fixed context xx, each rollout contains SS cells and can be represented by the empirical cell distribution Y=1S∑s=1SδysY=\frac{1}{S}\sum_{s=1}^{S}\delta_{y_s}. The policy therefore defines a distribution over these empirical populations, while the weighted target assigns probability wjw_j to each old-policy population.

What the outer MMD identifies. MMD-PO assigns reward-based weights to whole cell populations and trains the policy to match the resulting weighted distribution. The inner energy distance compares the cell distributions in two individual rollouts. The outer MMD compares which populations are generated and how frequently they occur. This goes beyond pooling all cells. For example, one model might generate populations containing only low-expression cells or only high-expression cells with equal probability, while another always generates populations containing an equal mixture of both. Their pooled cell distributions are identical, but their distributions over populations differ.

Our population kernel is characteristic: zero outer MMD implies equality of the distributions over unordered finite populations, beyond matching pooled cells or expression means. This follows because the kernel is Gaussian on the injective energy-distance embedding T(⋅)T(\cdot) (Appendix A.3), and a Gaussian kernel composed with a continuous, injective embedding into a separable Hilbert space is characteristic [1].

This characterization also clarifies the role of population size. The outer MMD identifies the distribution over empirical populations at the chosen size SS. At finite SS, differences between rollouts can reflect both differences in their underlying cell distributions and finite-cell sampling fluctuations.

For independent empirical populations P^S\widehat{P}_S and Q^S\widehat{Q}_S, each formed from SS i.i.d. cells sampled from underlying distributions PP and QQ, respectively, E[dE(P^S,Q^S)]=dE(P,Q)+a(P)+a(Q)S\mathbb{E}[d_E(\widehat{P}_S,\widehat{Q}_S)] = d_E(P,Q) + \frac{a(P)+a(Q)}{S}, where a(P)=EU,V∼iidP[∥U−V∥2]a(P) = \mathbb{E}_{U,V\overset{\mathrm{iid}}{\sim}P}[\lVert U-V\rVert_2] is the expected Euclidean distance between two independently sampled cells from PP, measuring within-population cell-state variability. a(Q)a(Q) is defined similarly.

Thus, increasing SS reduces the sampling contribution to the expected inner energy distance, while the underlying difference dE(P,Q)d_E(P,Q) remains.

To examine how this choice affects prioritization, we compared S∈{16,32,64}S\in\{16,32,64\} while keeping the other experimental settings unchanged. All three sizes achieve higher mean values than SFT on every reported ranking metric, with broadly similar AnDCG values across sizes. The prioritization gains persist throughout the tested range and are not specific to S=64S=64. The new results are updated in appendix F.2.

ConfigurationAnDCG-Rank 1%AnDCG-Rank 5%AnDCG-Geo 1%AnDCG-Geo 5%EF 1%EF 5%
SFT0.303 ± 0.0210.280 ± 0.0030.377 ± 0.0230.372 ± 0.0076.06 ± 0.534.50 ± 0.13
MMD-PO, S=16S=160.427 ± 0.0210.352 ± 0.0060.488 ± 0.0240.457 ± 0.0097.44 ± 0.255.36 ± 0.10
MMD-PO, S=32S=320.413 ± 0.0100.352 ± 0.0090.479 ± 0.0150.457 ± 0.0087.01 ± 0.625.54 ± 0.31
MMD-PO, S=64S=640.433 ± 0.0120.348 ± 0.0100.492 ± 0.0130.454 ± 0.0088.93 ± 1.455.74 ± 0.13

[1] Wynne, G., & Duncan, A. B. (2022). A Kernel Two-Sample Test for Functional Data. Journal of Machine Learning Research, 23(73), 1–51.

[W1/Q2]: Which connections between the practical surrogate-weighted MMD update and the original KL-regularized reward objective are mathematically exact, and which are heuristic?

Appendix A.1 establishes an exact equivalence between the original KL-regularized reward objective and reverse-KL fitting to the ideal target p⋆∝prefexp⁡(R/α)p^\star \propto p_{\mathrm{ref}}\exp(R/\alpha).

With a characteristic population kernel and a policy family capable of representing p⋆p^\star, exact global minimization of MMD recovers the same optimal distribution. This does not imply equivalent objectives or gradients. We replace reverse KL with MMD because it can be estimated from samples and differentiated through generated populations, avoiding the intractable policy likelihoods required for reverse-KL fitting.

The heuristic step in practical target construction is Eq. 7. We retain exponential reward weighting but replace the unavailable reference-to-old-policy likelihood-ratio correction with a reference-distance penalty. This defines a surrogate target favoring high-reward, reference-similar populations. The penalty is not an exact likelihood-ratio estimate, so the resulting target is not guaranteed to preserve the original KL-regularized optimum. Empirically, the gains in held-out perturbation prioritization over continued SFT and the evaluated policy-gradient baselines support the practical effectiveness of this surrogate update.

[W2/Q3]: Under matched training conditions, does nested MMD offer benefits over alternative distributional objectives, and do ranking performances require information beyond population means or first and second moments?

To address this concern, we added comparisons in which the outer MMD is replaced by other objectives, keeping the initialization, data, rollout budget, and optimization settings identical. We consider mean L2, and two full-distribution objectives, Wasserstein (W2) and Sinkhorn. The results are averaged across five different inference seeds.

MethodEF@1%, GT 10%EF@5%, GT 10%AnDCG-Rank@1%AnDCG-Rank@5%AnDCG-Geo@1%AnDCG-Geo@5%
SFT initialization6.06 ± 0.534.50 ± 0.130.303 ± 0.0210.280 ± 0.0030.377 ± 0.0230.372 ± 0.007
MMD-PO8.93 ± 1.455.74 ± 0.130.433 ± 0.0120.348 ± 0.0100.492 ± 0.0130.454 ± 0.008
Outer mean L27.66 ± 0.885.46 ± 0.040.394 ± 0.0110.321 ± 0.0060.433 ± 0.0160.412 ± 0.014
Outer Wasserstein5.84 ± 0.234.52 ± 0.050.374 ± 0.0130.298 ± 0.0040.459 ± 0.0170.422 ± 0.004
Outer Sinkhorn5.22 ± 0.184.41 ± 0.050.341 ± 0.0090.292 ± 0.0020.444 ± 0.0100.423 ± 0.003

MMD-PO achieves the best result on all six ranking metrics.

Sinkhorn and W2 also match the full distribution, yet both perform worse than mean matching and fall below the SFT initialization on EF. We attribute MMD’s advantage to its sample estimator, whose convergence rate does not depend on dimension, and to its smooth kernel gradients. Both properties matter when only a few rollouts are available per prompt, but optimal transport-based estimates are noisier and biased in this case.

Moreover, mean L2 matches only the mean of the rollout distribution, discarding variation across generated populations. MMD-PO outperforms it on all six metrics. Although the ranking score depends only on the predicted pseudobulk response, training with distributional matching improves it, indicating that the additional distributional signal benefits learning.

We thank the reviewer for the constructive feedback. We summarize the revisions addressing each concern below.

  • [W1/Q1] We formalized the policy as a distribution over empirical cell populations and clarified what the outer MMD identifies, why the population kernel is characteristic, and how finite-cell sampling affects the objective. New ablations with S = 16, 32, and 64 show prioritization gains across all tested population sizes.
  • [W1/Q2] We distinguished the exact KL derivation from the practical surrogate.
  • [W2/Q3] We added mean-L2, Wasserstein, and Sinkhorn comparisons under matched online training settings, reporting mean ± SD across five inference seeds. MMD-PO achieves higher mean values on all six ranking metrics. These results support the choice of fitting objective.

We appreciate your time and hope these additions address your questions and concerns. We are happy to clarify any remaining questions.

Comments on the paper

4 comments

Thank you very much for your positive comments and appreciation of our post-training interface, informative controls, and synthetic studies. Below are our responses to the specific queries raised.

W1 / Q1: Clarify the X-Cell sampling procedure and gradient computation.

Thank you for the question. X-Cell generates populations through four successive masked-reveal steps, with rollout variation arising from randomly sampled reveal orders.

Old-policy and reference populations are generated without gradient tracking (Algorithm 1, lines 5–9). Backpropagation occurs only during the current-policy update in line 15. Specifically, in Equation 9, gradients flow through the current-policy populations YiθY_i^{\theta}, while the old-policy populations YjoldY_j^{\mathrm{old}}, the weights wjw_j, and the kernel bandwidth remain fixed. The weights are computed from detached old-policy and reference predictions. The populations entering Equation 9 are generated by the same procedure used at inference.

Old-policy and reference rollouts are paired by generation index, sharing the same input context and reveal order. We have clarified these implementation details in the revised pseudocode (Algorithm 1).

W1 / Q2: Provide reproducible specifications for the synthetic experiments and Gaussian-GRPO baseline.

We thank the reviewer for this request, which has made the description of these experiments more complete. Appendix H.1 now fully specifies how the synthetic data are generated and used (added text in blue). Cell responses are produced by a fixed, known generative model, and a separate virtual-cell model is trained to predict them. The calibration set consists of 64 training perturbations, each with three replicate measurements of 64 cells.

We specify both noise components, namely errors shared across replicate measurements and errors that vary between measurements, including their distributions and scales. Noise-free responses are used for pretraining and to set the noise level. Evaluation uses 256 held-out perturbations whose responses never enter training or pathway selection.

We also give the exact rule for selecting a gene set using only calibration measurements. The rule favors gene sets with an average response near zero, large differences between perturbations, and high agreement between replicate measurements. Across 2,048 simulated problems, MMD-PO improves ranking on the selected gene set over the initial model by 0.0595 AnDCG@5% (95% CI: 0.0526 to 0.0664).

Because the ground-truth effects are known, these experiments test rankings against the true answers while varying feedback quality. They show that MMD-PO improves prioritization on unseen perturbations, and they provide a controlled setting for studying post-training from imperfect feedback.

Gaussian-GRPO baseline.

For both X-Cell and scGPT, Gaussian-GRPO uses the predicted gene-expression values as the means of a diagonal Gaussian policy. Actions are sampled from the old policy as

a=μold+σϵ,ϵ∼N(0,I),a = \mu_{\mathrm{old}} + \sigma\epsilon, \qquad \epsilon \sim \mathcal{N}(0, I),

with σ=0.1\sigma = 0.1, giving a per-gene variance of 0.010.01. The sampled actions are evaluated under the current and old policies, and the importance ratio is computed from mean rather than summed log-likelihoods:

rθ(a)=exp⁡(ℓˉθ(a)−ℓˉold(a))r_{\theta}(a) = \exp\left(\bar{\ell}_{\theta}(a) - \bar{\ell}_{\mathrm{old}}(a)\right)

ℓˉ(a)=1D∑d=1Dlog⁡N(ad;μd,σ2)\bar{\ell}(a) = \frac{1}{D}\sum_{d=1}^{D} \log \mathcal{N}\left(a_d; \mu_d, \sigma^2\right)

where DD is the total number of cell–gene coordinates (over cells and editable genes) included in the likelihood.

We use PPO ratio clipping with ϵclip=0.2\epsilon_{\mathrm{clip}} = 0.2 and regularize against the frozen SFT initialization with coefficient βKL=10−3\beta_{\mathrm{KL}} = 10^{-3}. The regularization term is the mean per-coordinate KL divergence between equal-variance Gaussian policies:

βKLD∑d=1D(μθ,d−μref,d)22σ2.\frac{\beta_{\mathrm{KL}}}{D} \sum_{d=1}^{D} \frac{\left(\mu_{\theta,d} - \mu_{\mathrm{ref},d}\right)^2}{2\sigma^2}.

For each method, hyperparameters were selected based on validation reward. The searches covered learning rate, Gaussian variance, KL regularization strength, and clipping thresholds, as applicable. Given the computational budget, we evaluated a limited set of configurations rather than an exhaustive grid. The reported Gaussian-GRPO configuration uses AdamW with learning rate 10−510^{-5}, effective batch size 64, and five training epochs. The full specification of Gaussian-GRPO is now provided in Appendix C.

Together, these additions provide complete specifications for reproducing both the synthetic experiments and the Gaussian-GRPO baseline.

W2 / Q3: Are the X-Cell gains stable across independent post-training runs, including the actual hits recovered for each pathway?

We thank the reviewer for this question. We acknowledge that the computational resources to repeat post-training across seeds for every experiment is limited. A single X-Cell post-training run uses 32 H200 GPUs for five epochs (Appendix D), and the full comparison covers three backbones, two datasets, and four methods. We instead repeated inference with five seeds for every trained checkpoint, and Table 1 in the revised PDF now reports mean ± SD over these runs.

DatasetBackboneGRPO / Flow-GRPOMMD-PO
HepG2X-Cell6.918 ± 0.5068.926 ± 1.448
HepG2scGPT3.168 ± 0.3824.583 ± 0.449
HepG2scDFM3.115 ± 0.8123.868 ± 1.856
T cellsX-Cell2.252 ± 0.0812.509 ± 0.115
T cellsscGPT1.341 ± 0.0821.460 ± 0.108
T cellsscDFM1.676 ± 0.0711.907 ± 0.059

MMD-PO has the higher mean EF@1% in all six settings.

Appendix H.2 provides a complementary breakdown of pathway-level AnDCG@5% improvements. To directly examine the HepG2 1% endpoint, we additionally report recovered hit counts and enrichment factors:

MethodTotal hits@1% / 148Macro EF@1%Pooled EF@1%
SFT initialization52.6 ± 3.46.06 ± 0.535.38 ± 0.34
Continued SFT53.0 ± 1.65.55 ± 0.595.42 ± 0.16
GRPO63.6 ± 1.76.92 ± 0.516.50 ± 0.17
MMD-PO69.0 ± 2.48.93 ± 1.457.06 ± 0.24

Total hits counts ground-truth hits recovered across 148 candidates from four perturbations for each of 37 pathways. Macro EF averages the enrichment factors calculated separately for each pathway. A pathway with few ground-truth hits receives a large EF from a single recovered hit, which explains its larger SD. Pooled EF divides the combined candidate hit rate by the overall ground-truth hit prevalence across all pathway–perturbation pairs.

On both total hits and pooled EF, which are insensitive to the small per-pathway shortlist, MMD-PO consistently outperforms every baseline. This confirms that its advantage reflects more true hits recovered overall.

W2: Do the ranking gains also demonstrate correct intervention direction?

Our primary prioritization task is to identify perturbations with the strongest pathway effects, whether activating or inhibiting. We therefore rank by absolute enrichment magnitude, treating strong effects in either direction as relevant candidates for follow-up. We agree that these metrics alone do not establish whether a perturbation acts in a particular desired direction.

We examine direction separately in Appendix H.2. The frozen-checkpoint HepG2 audit evaluates activation and inhibition rankings across all 37 retained Hallmark pathways. Figure 5B shows that MMD-PO improves AnDCG@5% over GRPO in both directions: +0.0258 for activation and +0.0233 for inhibition.

Q4: Which data were used for pathway and HVG selection, were held-out stimulated profiles excluded, and how were GSEA q-values calculated?

As described in Appendix B.2, the seven seed perturbations were selected from independent human genetic evidence. Six were chosen based on OMIM annotations linking them to T-cell activation–related immunodeficiencies, and KLF2 was chosen based on its loss-of-function burden association with lymphocyte percentage in UK Biobank. This initial seed-selection step did not use donor- or condition-specific experimental expression profiles. Held-out stimulated perturbed profiles were used for neither pathway selection nor HVG selection.

We calculate pathway enrichment using gseapy.prerank [1] on the pseudobulk perturbation-effect rankings, with 1,000 gene-label permutations. We use the package's built-in FDR q-values, calculated by comparing observed normalized enrichment scores with the permutation-derived null distributions across pathways within each perturbation. A perturbation–pathway pair is considered significant when q<0.05q < 0.05. The details are updated in Appendix A.5.

[1] Zhuoqing Fang, Xinyuan Liu, Gary Peltz. GSEApy: a comprehensive package for performing gene set enrichment analysis in Python. Bioinformatics, 2022, btac757. https://doi.org/10.1093/bioinformatics/btac757

Reviewer_3Q9fOpenReview ↗

Reply

Thank you for clarifying the X-Cell sampling procedure and synthetic experiments. The additional hit counts support the ranking improvement for the evaluated checkpoints. Please note in Table 1 that the SDs are across five inference seeds, not independent post-training runs. The revised X-Cell means imply EF@1% gains of 29.0% on HepG2 and 11.4% on T cells; please reconcile these with the 53.4% and 15.6% still reported in the abstract and main text. The revision addresses most of my concerns, and I maintain my rating of 3 (weak acceptance).

Thank you for the constructive feedback and positive assessment. We are glad the revisions address most of your concerns. We have updated the abstract and main text to report the mean gains. We will also check the full manuscript for consistency in all reported means, SDs, and relative improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.