acceptodds
Under review as a conference paper at ICLR 2027

Why Do Sophisticated Perturbation Prediction Models Not Outperform Learning-Free Baselines?

Abstract

Why are modern perturbation-prediction models often not much better than simple learning-free baselines? This phenomenon has been widely observed in genetic perturbation benchmarks. However, its interpretation in chemical perturbation prediction remains unclear. Using sci-Plex3 as a primary case study, we show that simple pseudo-bulk-level baselines, together with two new single-cell-level learning-free baselines, remain highly competitive, even against modern diffusion-based models. We then propose a three-axis diagnostic framework comprising Input Use, Response Scaling, and Stratified Performance, which evaluates whether perturbation information is used, whether predicted responses scale with the strength of the observed cellular responses, and where along the perturbation effect distribution models succeed or fail. Applying the framework to ChemCPA, CondOT, Doloris, and PerturbDiff reveals that failure to outperform learning-free baselines can arise from two distinct causes: limited perturbation signal in the data or limited utilization of that signal by the model. Independent evaluation on Tahoe-100M further confirms that the competitiveness of learning-free baselines is dataset- and metric-dependent: ChemCPA remains below the perturbation reference, while PerturbDiff exceeds it on all pseudo-bulk metrics but not consistently on distributional metrics. Our results suggest that comparing performance against learning-free baselines alone is not a reliable indicator of model quality. Instead, our diagnostic analyses provide mechanistic insights into why models succeed or fail, help distinguish data limitations from model limitations, and guide the development of more robust perturbation-prediction models.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
single cell biology, perturbation prediction, benchmark, learning-free baselines, chemical perturbation, scRNA sequencing, personal treatment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EHutOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper benchmarks three transport-based models (CondOT, Doloris, and PerturbDiff) against a latent-disentanglement baseline (ChemCPA) and simple linear baselines on two chemical perturbation datasets, evaluating generalization to unseen drugs. It proposes an inference-time diagnostic that quantify how much a model's predictions depend on the perturbation identity (input usage). The paper also evaluates response scaling, defined as the slope of an OLS regression of predicted on true per-gene expression changes, together with coeficient of determination (R2\textrm{R}^2) stratified by effect size. It claims that models that make use of perturbation information, notably PerturbDiff, achieve competitive response scaling and improved R2\textrm{R}^2 on high-effect perturbations relative to the linear baselines. Finally, an ablation on ChemCPA that adds conditional sampling and a perturbation-induced signal is reported to improve it on all three axes: input usage, response scaling, and effect-size-stratified performance.

Critical strengths
  • The paper benchmarks deep learning models and introduces a simple inference-time diagnostic for perturbation input usage. It ties low input usage to why these models can fail to outperform simple baselines.
  • The paper proposes a fix for ChemCPA that improves perturbation usage and shows gains on all three axes: input usage, response scaling, and effect-size-stratified performance.
Critical weaknesses

Novelty and contribution The paper claims to improve on several previously proposed benchmarks by focusing on chemical perturbations. The approach and conclusions appear to be a straightforward extension of those benchmarks, with conclusions consistent with those for genetic perturbations. Simple baselines remain competitive, even on the chemical perturbation prediction task, which limits the significance of the findings.

Motivation and redundancy of the proposed diagnostics

  • The proposed diagnostics (input usage, the β\beta coefficient, and stratification by effect size) can largely be captured by existing metrics. Input usage: it is unclear why the perturbation discrimination score (PDS) is not used, since it could capture input usage more cheaply. β\beta coefficient: the motivation is unclear, since β\beta is already highly correlated with R2\textrm{R}^2. It is also uninformative for noisy predictions, which can still attain a high β\beta.

Experimental design and evaluation

  • Cross-validation: the main paper appears to report a single fold of the 80/20 split. It is unclear why the 5-fold results are not consolidated and reported in the main text.
  • DEG selection: the fixed top-50 DEG selection could be strengthened by using a Benjamini–Hochberg FDR threshold, since some perturbations may have fewer than 50 significant DEGs.
  • Metrics: it is unclear why DES recall is not included in the evaluation.

Baselines and completeness The paper proposes cell-level linear baselines (scControl and scPerturb) but never benchmarks against them. Not all tables and results include all linear baselines and deep learning models, which makes comparisons inconsistent.

Reporting and statistical rigor

  • Results are spread across several tables. A single table for the 5-fold setup, including all metrics (with β\beta, input usage, and the stratified results), would be clearer.
  • The paper should consistently report the mean and standard deviation across all metrics. These appear to overlap substantially between the best-performing deep learning models and the linear baselines, which weakens the claimed differences.
  • Figure 2: the error bars overlap substantially, making the figure difficult to interpret. Effect strength could be reported over quantiles (instead of 6 bins) as numerical values.

ChemCPA modification and ablation

  • The ChemCPA setup needs further explanation. It is unclear what the conditional sampling approach is and how the auxiliary perturbation loss is included.
  • Table 5: the ChemCPA results are mixed. For instance, higher input usage does not translate into higher R2\textrm{R}^2. The gains appear only on the proposed response scaling metric and on high-effect perturbations. This is somewhat circular, since the improvement shows up mainly on the metrics the paper itself introduces, and it weakens the claim of improvement on all three axes.
Questions for authors
  • Could you report the perturbation discrimination score (PDS) for all models and compare it with the proposed input-usage diagnostic?
  • Could you clarify what the β\beta coeffienct metrics adds beyond R2\textrm{R}^2?
  • Could you extend the ChemCPA results (Table 5) to include all linear and deep learning baselines under the same set of metrics, so that the effect of the modification can be compared directly with the other methods? Also, could you explain why, and show whether the improvements hold on metrics other than the proposed ones (PDS, DES recall)?
  • Could you describe the conditional sampling approach and how the auxiliary perturbation loss is incorporated?
  • What conclusions here are specific to chemical perturbations, as opposed to extending those already found for genetic perturbations?
8 comments
AuthorsOpenReview ↗

Response to Reviewer EHut (1/3)

We thank the reviewer for the thorough review, which led to a more complete and more consistent evaluation. We summarize the major changes first and respond to each of your questions in more detail after this:

  • W1 / Q5 (novelty; what is specific to chemical perturbations). We have clarified the positioning, contribution and significance of our work against the six references supplied by Reviewer KSCd\color{blue}{\textbf{KSCd}} (App. A, Table 6). A tie with a learning-free baseline can be a limit of the data or of the model; the diagnostics tell the two apart, and which one it is changes what to do about it.
  • W2 / Q1 / Q2 (redundancy with existing metrics). PDS is now reported for every model (Table R1); β and R² differ by construction, not only empirically (App. C.3).
  • W3 (evaluation design). Five folds for every method except Doloris (three), plus a BH-FDR gene set and DES recall (revised Table 1).
  • W4 (baselines and completeness). scControl and scPerturb are labelled rows of Table 1, which lists every model and reference on sci-Plex3.
  • W5 (reporting and statistics). Two consolidated five-fold tables, mean ± standard deviation over folds, and a redrawn Fig. 2 of paired differences.
  • W6 / Q3 / Q4 (ChemCPA variants). Described in App. B.3 and reported on all metrics; the sampler improves the standard accuracy metrics, the directional loss trades aggregate accuracy for compound use (Table E3).

W1 / Q5. Novelty, significance, and what is specific to chemical perturbations

1. "A straightforward extension of those benchmarks."

We agree that this is an active field and that a tie with simple baselines is a common observation; our work is indeed inspired by those benchmarks. As our positioning table shows (Table K1 in our reply to Reviewer KSCd\color{blue}{\textbf{KSCd}}), our main contribution is to understand why the tie happens, what it indicates, and how it can be fixed. Existing benchmarks mostly propose a better metric or a better model. Only Systema (Viñas Torné et al., 2025) studies the cause systematically from the data perspective, and it does so for genetic perturbations. Chemical perturbation data have a different nature: a compound is applied across doses and may do nothing at one dose and a great deal at another (Srivatsan et al., 2020), so the tie has to be resolved per condition: where the data leave no room it is expected, and where they do it is a limit of the model. None of the six works substitutes the model's own perturbation input or regresses predicted magnitude on observed. This is also our answer to Q5.

2. "Simple baselines remain competitive, which limits the significance."

That is our starting point, not our main finding. Where effects are weak little is available to win: 60% of held-out conditions have fewer than 10 significant genes (BH-FDR 5%), and a split-half reproducibility ceiling (the technical duplicate of Miller et al., 2025) sits at most 0.016 DEG50 R² above the per-dose perturbation mean. Where they are strong the gap belongs to the model: PerturbDiff exceeds the per-dose mean by 0.378 in the strongest bin, while models with similar aggregate DEG50 R² depend far less on the compound (Table R1). And it is actionable: ChemCPA's sampler alone raises DEG50 R² from 0.732 to 0.786.

AuthorsOpenReview ↗

Response to Reviewer EHut (2/3)

W2 / Q1 / Q2. Are the diagnostics captured by existing metrics?

1. Input Use and PDS (Q1).

Thank you for the suggestion; PDS is now reported for every model (Table R1; chance 0.5). PDS compares predictions with observations, so a low score does not say whether the model ignored the compound or used it and got it wrong. Input Use intervenes on the compound input of the trained model at inference time, which separates those two cases. The two can disagree: on Tahoe-100M, ChemCPA reaches a PDS of 0.635, above the compound-blind per-dose mean (0.611), yet swapping the compound changes its prediction by 0.000 (Table K2 in our reply to Reviewer KSCd\color{blue}{\textbf{KSCd}}).

2. What β adds to R² (Q2).

We thank the reviewer for the interesting question. There is a real link in general, and it is worth stating precisely: for a single regression of yy on xx, R2=β2 Var(x)/Var(y)R^2 = \beta^2\,\mathrm{Var}(x)/\mathrm{Var}(y), so the slope and the R2R^2 of that regression carry the same information up to a variance ratio. If the R2R^2 we report were the R2R^2 of β's own regression, β would indeed be redundant. It is not. The two are computed on different data points: for R², a data point is a gene; for β, it is a condition:

One data pointComputed over
R²a genethe genes of one condition (full panel or DEG50), then averaged over conditions
βa conditionthe held-out conditions

Because the two run over different things, one can be high while the other is zero: a predictor that reproduces the baseline and none of the response still explains most of the variance in expression levels, yet has β exactly 0. That is the control mean (Table E2). β is also a slope rather than a correlation, so it registers systematic shrinkage: a prediction can order conditions perfectly and still sit well below β = 1, which is the second row of Table E2.

Table E2. R² and β measure different things (sci-Plex3, five folds). Controlled predictors, not models.

PredictorDEG50 R²β
Control mean (no response)0.743 ± 0.0820.00
True response × 0.3 (shrunk)0.874 ± 0.0400.30

3. "β is uninformative for noisy predictions."

We agree that β alone can stay high for a noisy prediction as it measures the size of the response, not its direction or which genes move. That is why we report it alongside DES recall and PDS. On this dataset the three agree about which models use the compound: PerturbDiff is highest on all three and CondOT lowest (Table R1).

Full results. App. C.3: the controlled predictors with DEG50 and full-panel R² and β.

W3. Single fold; top-50 vs BH-FDR; DES recall

Thank you for these suggestions; all three are implemented (Table R1 in the general response; revised Table 1).

1. "A single fold of the 80/20 split."

Every method is now reported on five folds in the main text, except Doloris, which ran on three: on folds 1 and 3 its mask model collapsed with each of three random seeds, and recovering them would have required fold-specific tuning that no other method received (App. B.2).

2. "A Benjamini–Hochberg FDR threshold."

Thank you for the suggestion. We followed ChemCPA to obtain the DEG50 for the sci-Plex3 dataset and STATE for the Tahoe-100M dataset. However, we agree this is an interesting point, so we added FDR-set R2R^2, computed on each condition's significant genes at BH-FDR 5%. We observe that only 723 of 1,788 conditions have ten or more significant genes, and the median is one to two in the three weakest effect-size bins, so for most conditions a fixed top-50 panel is larger than the set of genes that actually move. The conclusions are unchanged on the FDR set: PerturbDiff 0.56 against 0.38 for the per-dose perturbation mean.

3. "DES recall is not included."

Added for every method in Table 1; PerturbDiff is the highest in every fold.

W4. scControl / scPerturb; incomplete tables

Thank you for pointing this out, we have clarified the notation of the method: scControl and scPerturb were benchmarked, but their values sat in Table 1's single-cell columns under the pseudo-bulk row names. Table 1 now labels both, and revised Tables 1 and 5 list every model and every reference on sci-Plex3.

AuthorsOpenReview ↗

Response to Reviewer EHut (3/3)

W5. Scattered tables; overlapping ±; Fig. 2; quantiles

Thank you for these suggestions on reporting; we have followed each. Two five-fold tables now hold everything: Table 1 the accuracy metrics, Table 5 the three diagnostics together with the stratified results.

1. "Mean and standard deviation overlap between models and baselines", and "Fig. 2: the error bars overlap."

The revision reports mean ± standard deviation over folds throughout, and Fig. 2 now plots each method's paired difference to the per-dose perturbation mean. We do not claim a difference in aggregate accuracy between the models. The aim is to explain why they score so close to the learning-free baselines, and the diagnostics of Table 5, where the methods do differ, are what address that.

2. "Effect strength over quantiles, as numerical values."

The six bins are already equal-count quantiles of effect size, and App. D reports every per-bin value numerically.

W6 / Q3 / Q4. The ChemCPA variants

1. "What is the conditional sampling approach, and how is the auxiliary loss included?" (Q4)

Thanks for your question. The description is now in App. B.3. Condition-balanced sampling draws 64 training conditions uniformly per batch, then 32 cells within each. The directional loss adds cosine and magnitude terms on per-condition residuals to the per-dose training mean. Both settings were selected on a validation partition built from the training drugs only.

2. "The gains appear only on the metrics the paper introduces." (Q3)

Thank you for the question. Condition-balanced sampling improves three metrics we did not introduce: DEG50 R² 0.732 to 0.786, full-panel R² 0.884 to 0.922, FDR-set R² 0.291 to 0.380 (Table E3). The directional loss also improves DES recall, which you suggested, from 0.029 to 0.038.

3. "Higher input usage does not translate into higher R²", and improvement on all three axes.

Thank you for the question! The sampler improves accuracy and compound use together, while the directional loss raises compound use further (Input Use −0.041 to −0.075, β 0.15 to 0.22) at a small cost in aggregate accuracy (full-panel R² 0.865 against 0.884). The aggregate averages two regimes: in the weak bins the ceiling leaves at most 0.016 above the per-dose perturbation mean, so a model that uses the compound has nothing to win there and pays a little instead, while in the strongest bin both variants move from 0.012 below that reference to about 0.115 above it. PerturbDiff shows the same pattern.

Table E3. ChemCPA and its variants (five-fold means). Strongest bin: DEG50 R² minus that of the per-dose perturbation mean, in the last of six effect-size bins.

ChemCPADEG50 R²Full R²FDR-set R²DES recallΔR² (T2)β (full)Strongest bin
Original0.7320.8840.2910.029−0.0200.104−0.012
+ condition-balanced sampling0.7860.9220.3800.028−0.0410.153+0.115
+ sampling + directional loss0.7470.8650.2880.038−0.0750.224+0.116

4. "Extend Table 5 to all baselines under the same metrics." (Q3)

Thank you for your suggestion! We have revised Tables 1 and 5 to include both variants.

We hope these changes address your concerns. If any point remains open, please tell us and we will answer it.

Reviewer_JgZsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper asks why chemical perturbation models barely beat "learning-free" references (control and perturbation mean) on 30 unseen sci-Plex3 drugs. It proposes three diagnostics: "Input Use" (corrupt the drug embedding, Eq. 1), "Response Scaling" (slope β\beta of predicted on observed effect norm δ\delta, Eq. 2-3) and "Stratified Performance" (DEG50 R2R^2 in six δ\delta bins).

Critical strengths
  • I like the question. The drug swap (T2) and the effect-size bins are cheap and show that Tab. 1 hides how in the strongest bin PerturbDiff sits roughly 0.5 to 1.1 DEG50 R2R^2 above the control mean (read off Fig. 2). A swap drop of 0.143 vs 0.000 (Tab. 2) is hard to explain by noise, so I buy that PerturbDiff uses the drug while ChemCPA and CondOT (at least as run here) mostly don't.
  • Condition-balanced sampling alone lifts ChemCPA from 0.739 to 0.810 DEG50 R2R^2 and the strongest bin from -0.095 to 0.248 at no cost in the weakest (Tab. 5). It's a simple fix, and the one place where the attribution is tested by intervention.
Critical weaknesses
  • The "data limitation" half of the attribution is never tested. "Limited signal" (L399-401) is inferred from the skew, but with no estimate of what any predictor could reach per bin. The main text also skips a model-side cost. PerturbDiff sits roughly 0.02 to 0.1 below the control mean in the four weakest bins of every cell line (Fig. 2 insets) and has the lowest full R2R^2 in Tab. 1 (0.866, perturbation mean 0.920). It's small (by my reading 0.03 of the aggregate, vs 0.12 gained in the top bins) but weak signal explains a tie but not a loss. The references pool all doses (L151-152) while the models get the dose (L146). T2 changes only the compound (L216-217), so the swap result stands, but the margins and β\beta may not! And δ\delta is an uncorrected norm of a difference of sample means (Eq. 2), so few cells inflate it (strong-effect conditions have fewer cells).
  • The evidence is thin for an explanatory claim imo, only one fold shown (L284-285), no intervals on ΔR2\Delta R^2 and β\beta (Tab. 2, 3, 5), and top-bin gains that rest on 20 conditions per cell line. As far as I know PerturbDiff and CondOT ship no unseen-drug sci-Plex3 setup. Tahoe gets no diagnostic, and the "more visible" advantage predicted at L410-412 is mixed.

Overall useful tools and a model split that holds up, so I lean accept. But the "data limitation" reading needs folds, intervals, a ceiling and a dose-aware reference before I'd believe it.

Questions for authors
  1. (experiment) Could you add a per-cell-line, per-dose training mean to Tab. 1, 3 and Fig. 2, and report the T2 drop per bin?
  2. (experiment) How do Fig. 2 and Tab. 3 look with a noise-corrected δ\delta (e.g. disjoint half-samples), and where does a technical-duplicate ceiling (Miller et al., 2025) sit per bin? If the top-bin lead survives Q1 and the ceiling leaves little room in the weak bins, I'm comfortable with acceptance. If the lead goes away, I'd go down.
  3. (clarification) Is the ordering in Tab. 1-3, including ΔR2\Delta R^2 and β\beta, stable over the five folds for all four models?
  4. (clarification) On which data were the adapted models and the Tab. 5 variants tuned, and were the 30 test drugs kept out of every selection?
6 comments
AuthorsOpenReview ↗

Response to Reviewer JgZs (1/3)

We thank the reviewer for the precise and constructive review, and for stating a clear condition for acceptance. We summarize the major changes first, then respond to each comment in detail:

  • W1a / Q2 (ceiling, noise-corrected δ). Both are added: the strongest-bin lead survives, and the ceiling leaves little room in the weak bins (App. D; Table J1).
  • W1b (model-side cost). Now reported and analysed: PerturbDiff is 0.068 below the per-dose mean in the weak bins (Sec. 5.3; App. D).
  • W1c / Q1 (dose-aware reference, T2 per bin). The per-dose mean is used throughout; the T2 drop is concentrated in the strongest bin (Table J2; App. C.4).
  • W2 / Q3 (folds and intervals). Five folds with ± over folds: the ordering on the diagnostics is stable, and the five-fold means are those reported in the paper.
  • Q4 (tuning data). The test drugs entered no selection step (App. B.2).

W1a / Q2. Noise-corrected δ̄ and a technical-duplicate ceiling Thanks for raising this really interesting point. Following your suggestion, we ran both tests as specified, and both parts hold: in the strongest bin PerturbDiff's lead over the per-dose perturbation mean survives in 5/5 folds (+0.378), and in bins 1–3 the ceiling leaves at most 0.016 above that reference (Table J1).

δˉ\bar{\delta} is re-estimated from two disjoint half-samples, which is used for every stratified result. Sec. 3.2 now states that the pseudo-bulk effect size, as the norm of a difference between sample means, is inflated when a condition has few cells. We also follow Miller et al. (2025) to report the technical duplicates (one half of a condition's cells scored against the other) and the noise-corrected ceiling it implies.

Table J1. DEG50 R² per effect-size bin (1 = weakest), five folds. Mean ± standard deviation over the five folds. Lower four rows: per-dose mean − model (negative: the model is above). ∗: the model differs significantly from the per-dose mean (95% bootstrap confidence interval over drugs). Bold: the model is significantly above the per-dose mean.

Bin123456
Technical duplicate0.964±0.0050.965±0.0030.966±0.0050.966±0.0020.958±0.0060.905±0.036
Ceiling (rescaled)0.991±0.0010.991±0.0010.991±0.0010.991±0.0000.989±0.0010.976±0.009
Per-dose mean0.980±0.0050.977±0.0020.975±0.0030.962±0.0080.879±0.063−0.104±0.416
Room = ceiling − per-dose mean+0.010±0.004+0.014±0.001+0.016±0.003+0.030±0.008+0.110±0.062+1.080±0.408
Per-dose mean − model
ChemCPA+0.051±0.048∗+0.049±0.044∗+0.046±0.034∗+0.045±0.040∗+0.062±0.050∗+0.012±0.079
CondOT+0.004±0.001∗+0.005±0.001∗+0.006±0.002∗+0.004±0.002∗+0.009±0.008∗+0.023±0.050
Doloris (3 folds)+0.072±0.006∗+0.066±0.004∗+0.069±0.000∗+0.076±0.003∗+0.045±0.040∗−0.232±0.107∗
PerturbDiff+0.075±0.027∗+0.076±0.014∗+0.058±0.007∗+0.062±0.013∗+0.033±0.011∗−0.378±0.125∗

Fig. 2 is redrawn. It now shows each method's paired difference to the per-dose perturbation mean with the standard deviation over folds, for bins 1–4 pooled, bin 5 and the strongest bin, and the grey band gives the room between that reference and the ceiling, i.e., the dose-aware reference of Q1, the intervals and the attainable headroom in one panel.

Tab. 3 is redrawn. It reports β (both noise corrected and raw) for every method. The values shrink and the ordering is unchanged: CondOT 0.048 → 0.040, ChemCPA 0.104 → 0.063, Doloris 0.241 → 0.173, PerturbDiff 0.408 → 0.376, with the per-dose mean at 0.044. PerturbDiff stays highest in 5/5 folds.

The strongest-bin lead is also unchanged on the raw effect-size bins (0.379) and with every condition subsampled to 50 cells (0.387; Table K3 in our reply to Reviewer KSCd).

Full results. App. D: per bin, cell numbers, effect size relative to sampling noise, significant genes, the ceiling, and each model's distance to the per-dose mean and to the ceiling.

AuthorsOpenReview ↗

Response to Reviewer JgZs (2/3)

W1b. Weak signal explains a tie, not a loss

You are right that weak signal cannot explain a loss. Sec. 5.3 now reports this cost, and App. D quantifies it (0.068 below the per-dose perturbation mean in bins 1–4).

W1c / Q1. Per-cell-line, per-dose reference; T2 per bin Thank you for your suggestions! We have added the per-cell-line, per-dose reference and analyzed T2 per effect-size bin. The per-dose reference is cell-line specific like the other two, and is the strongest of the three (DEG50 R² 0.776, against 0.768 dose-pooled per-cell-line perturbation mean and 0.743 control); it is now in Tables 1 and 3 and in Fig. 2, and every margin we report is measured against it. Against this reference PerturbDiff's strongest-bin lead is 0.378, and the ordering holds in 5/5 folds (Table J1). β\beta moves the same way: the per-dose reference reaches 0.044 on the full panel where the control and dose-pooled means sit at 0.000 and −0.001, and against that PerturbDiff reaches 0.408 while CondOT, the least compound-sensitive model, sits at 0.048 (Table R1 in our general response; Table 3).

The T2 drop per bin is in Table J2: the swap costs PerturbDiff 0.413 ± 0.127 in the strongest bin and 0.01–0.02 in bins 1–4, in every fold. Weak responses are nearly the same across compounds, so a swap changes little; strong responses are distinct, so it harms most.

Table J2. ΔR² (T2) per effect-size bin (sci-Plex3, five folds; 1 = weakest). Change in DEG50 R² when the compound embedding is swapped. Mean ± standard deviation over folds. Bold: largest drop per bin.

Bin123456
CondOT−0.001±0.001−0.001±0.0010.000±0.001−0.001±0.0010.000±0.001+0.002±0.008
Doloris (3 folds)−0.004±0.002−0.003±0.001−0.002±0.004−0.005±0.004−0.014±0.009−0.140±0.057
PerturbDiff−0.015±0.009−0.012±0.008−0.021±0.017−0.022±0.017−0.054±0.025−0.413±0.127
ChemCPA+0.006±0.017+0.002±0.0060.000±0.002−0.006±0.010−0.011±0.007−0.099±0.038

Full results. App. C.4: the same table with both ChemCPA variants.

AuthorsOpenReview ↗

Response to Reviewer JgZs (3/3)

W2 / Q3. Folds, intervals, top-bin support, adapted models, Tahoe

Thank you for these points; we address each in turn.

1. "Only one fold shown"; Q3, is the ordering stable for all four models? Everything is now on five folds, except Doloris, which ran on three (see the note to Table J3), and the five-fold means are those reported in Tables 1, 3 and 5. On the diagnostics the top of the ordering is stable: PerturbDiff has the largest T2 drop and the highest β in 5/5 folds (Table J3). Rank changes across folds occur only among ChemCPA, Doloris and CondOT, which all make limited use of the compound, so their relative order carries little information.

2. "No intervals on ΔR² and β." Added throughout: mean ± standard deviation over folds in Tables 1, 3 and 5, and in Tables J1–J3 here.

3. "Top-bin gains rest on 20 conditions per cell line." With five folds the strongest bin holds 300 conditions from 77 drugs, and PerturbDiff's lead holds in every fold (Table J1).

4. "PerturbDiff and CondOT ship no unseen-drug sci-Plex3 setup." That is correct. App. B.2 describes our adaptations in detail, and the results are properties of those configurations.

5. "Tahoe gets no diagnostic, and the 'more visible' advantage is mixed." The three diagnostics are now run on Tahoe-100M over five folds (Table K2 in our reply to Reviewer KSCd), where PerturbDiff again has the larger T2 drop and the higher β. By "more visible" we meant the pseudo-bulk metrics: PerturbDiff exceeds the perturbation mean on all four of them on Tahoe-100M, against two of four on sci-Plex3. On the single-cell metrics the cell-level baselines we propose remain hard to beat on both datasets, and Sec. 5.4 now says so.

Table J3. Per fold (sci-Plex3). ΔR² (T2): change in DEG50 R² when the compound embedding is swapped. Bold: best in the fold. Doloris: folds 0, 2 and 4; on folds 1 and 3 its mask model collapsed to a single cell line with each of three random seeds, and recovering them would have required fold-specific tuning that no other method received (App. B.2), so we report the mean and std across three folds for Doloris and across five folds for all the other methods.

Fold 0Fold 1Fold 2Fold 3Fold 4Mean ± sd
ΔR² (T2)
CondOT0.0000.0000.0000.0000.0000.000 ± 0.000
Doloris−0.047–−0.044–−0.015−0.035 ± 0.015
PerturbDiff−0.154−0.097−0.124−0.071−0.064−0.102 ± 0.033
ChemCPA−0.027+0.009−0.036−0.006−0.039−0.020 ± 0.019
β (full panel)
CondOT0.0430.0300.0440.0610.0630.048 ± 0.012
Doloris0.269–0.245–0.2090.241 ± 0.025
PerturbDiff0.6180.6510.4170.1060.2470.408 ± 0.210
ChemCPA0.0340.0760.2520.0230.1330.104 ± 0.084

Full results. Revised Tables 1 and 5: mean ± standard deviation over folds for every model and reference; App. E.2 and E.3: the same for the ChemCPA arms, on the pseudo-bulk and distribution-level metrics.

Q4. Tuning data

Thank you for the question. No held-out drug entered any selection step (as we stated in App. B.2). The two variants' sampler and loss settings were chosen on a validation partition of the training drugs only (seen drugs, unseen conditions). The sensitivity table in App. B.3 shows the test-set performance of one split for reference; the settings were chosen on the validation partition.

We hope these changes address your concerns. If any point remains open, please let us know, and we will try our best to answer it.

Reviewer_KSCdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors investigate why sophisticated perturbation models often fail to beat simple, learning-free baselines. Four models are evaluated, primarily on sci-Plex 3, using three diagnosits: (1) Input use, (2) Response scaling, (3) Stratified performance. They find that models can differ substantially on how perturbation information is utilized, and that some outperform baselines on perturbations exhibiting strong effects. Additionally, they find that an additional loss can improve some aspects of ChemCPA's performance.

Critical strengths

1. Empirical analysis.

The authors provide an interpretable analysis of perturbation prediction behavior beyond aggregate metrics. They highlight differences in how models utilize perturbation-specific information.

2. Training improvements.

The experiments with condition-balanced sampling demonstrate that relatively simple training modifications can improve perturbation prediction.

Critical weaknesses

1. Narrow benchmarking and overlap with existing work.

The paper claims that existing benchmarks have primarily concerned themselves with comparing models instead of asking the question of why they fail. I believe this characterization to be inaccurate; all three diagnostics have substantial overlap in existing literature.

Input Use: Systema [1] investigate whether perturbation models reproduce shared responses or are able to capture perturbation-specific ones.

Response scaling: PRESAGE [2] have explicitly evaluated magnitude and direction of predicted perturbation effects separately. scDrugPerturb-Bench [3] analyze biological-response fidelity and recovery of effect size specifically for chemical perturbation.

Stratified Performance: Li et al. [4] report prediction performances as a function of perturbation effect strength, including within datasets. They report systematic underestimation of perturbation responses. Schäfer et al. [5] further develop a framework for evaluating perturbation-specific effects and assessing the behavior of evaluation protocols.

Additionally, Heidari et al. [6] study evaluation failures in both genetic and chemical perturbation prediction, including sci-Plex3, using controlled experiments and simulations.

These works go beyond comparisons, and many are already cited in the submission without sufficiently distinguishing their contributions. The precise implementation and combination of the proposed diagnostics may be new, however, several of the authors methods and conclusions seem well established in prior work.

2. Insufficient validation and potentially confounded conclusions.

The diagnostic framework is comprehensively evaluated primarily on sci-Plex 3. Although Tahoe100M is included, the three diagnostics are not systematically replicated there. This limits the generality of the conclusions, in particular given the vast variation in perturbation responses across datasets.

Additionally, the distinction between data limitations and model limitations is not convincingly established. Effect magnitude is estimated from pseudobulk differences and is therefore sensitive to sample noise. A strong negative correlation is reported between effect magnitude and cell count (Spearman = -0.761, Figure 4). This raises concerns that some estimated strong effects could reflect increased sampling noise rather than biological signal.

Lastly, the diagnostics the authors propose assess perturbation sensitivity and magnitude without necessarily establishing that the predicted gene-level responses may be biologically correct. Thus, the claimed mechanistic explanation seems stronger than the evidence supports.

References

[1] Vinas Torné et al. (2025). Systema: A Framework for Evaluating Genetic Perturbation Response Prediction Beyond Systematic Variation.

[2] Littman et al. (2025). Gene-embedding-based Prediction and Functional Evaluation of Perturbation Expression Responses with PRESAGE.

[3] Lehang Li et al. (2026). A Mechanism-Annotated Benchmark Reveals Limited Fidelity to Drug-Response Signatures in Single-Cell Perturbation Models

[4] Lanxiang Li et al. (2026). A Systematic Comparison of Single-Cell Perturbation Response Prediction Models.

[5] Schäfer et al. (2026). Towards Principled Evaluation of Single-Cell Perturbation Prediction Models.

[6] Heidari et al. (2026). Evaluating Single-Cell Perturbation Response Models Is Far from Straightforward.

Questions for authors
  1. Novelty clarification: Could the authors clarify how each proposed diagnostic differs from the corresponding analyses in related work ([1], [2], [4], [8])? What additional insights does the framework enable?

  2. Do the stratified results remain consistent after accounting for the relationship between cell count and uncertainty in estimated perturbation effect magnitude? For example through independent cell subset or noise correction?

  3. Generalizability: Can the authors apply the diagnostics to Tahoe100M to establish whether their conclusions hold across datasets with different perturbation effect distributions?

6 comments
AuthorsOpenReview ↗

Response to Reviewer KSCd (1/3)

We thank the reviewer for the constructive suggestions, which have helped us improve the paper. We summarise the main changes first and respond to each question in detail below:

  • W1 / Q1 (overlap with prior work). All six works are cited and positioned in a new table (App. A; Table K1).
  • W2a / Q3 (generality on Tahoe-100M). The three diagnostics are reported on five folds and every conclusion from sci-Plex3 holds (App. G; Table K2).
  • W2b / Q2 (cell count and noise). Effect size is corrected for sampling noise and the analysis repeated at equal cell number; the stratified results hold (App. D.3; Table K3).
  • W2c (biological correctness). DES and PDS are added; PerturbDiff is the highest on both (Table R1).

W1 / Q1. Relation to [1]–[6]

We thank the reviewer for the suggestions. We have cited all six works and clarified our contribution in the revised manuscript (Sec. 1; App. A, Table 6).

Our position: Many works observe that learned models tie with simple baselines. We ask why the tie happens and what to do about it. Existing benchmarks mostly answer with a better metric or a better model; only Systema [1] studies the cause systematically from the data perspective, and it does so for genetic perturbations. For chemical perturbations the answer depends on the condition, because a compound can do nothing at one dose and a great deal at another. We therefore ask three questions of the same trained models, per condition (drug × dose × cell line).

How each diagnostic differs (Q1)

  • Input Use: does the model use the compound? Earlier work compares a model with a perturbation-blind baseline and infers the answer from the predictions [1]. We test it directly: we give the trained model the wrong compound, and if the prediction barely moves, the model does not use the compound. This matters for model design: if the compound is not used, a better drug representation cannot help, and the training has to change first.
  • Response scaling: does the predicted effect grow with the true effect? PRESAGE [2] scores a yes/no label (effect or no effect), scDrugPerturb-Bench [3] a ratio within each case, and Lanxiang Li et al. [4] a rank correlation. None of these says how much the prediction grows from weak to strong conditions. This matters in practice: choosing which compounds and doses to follow up depends on how strong the predicted response is. Our slope β measures it: β below 1 means that strong effects are underestimated.
  • Stratification: is there room to do better? [5] and [6] build an attainable reference but do not relate it to effect size; [4] relates accuracy to effect size but gives no attainable reference. We put a performance ceiling inside each effect-size bin. Weak conditions can dominate a dataset, and in the three weakest of six bins the ceiling is at most 0.016 above a compound-blind mean, so a model has almost nothing to gain there, however strong it is.

What it enables (Q1)

Together, the three answers say what a tie means for a given condition. Where there is no room, the tie is expected. Where there is room and the model does not use the compound, the model is the limit. This makes each result actionable: fix the metric, fix the model, or stop where the data leave no room. On ChemCPA the diagnosis pointed at training rather than at the drug representation (App. C.2), and changing only the sampler improves accuracy (Tables 1 and 5). None of [1]–[6] gives this reading (Table K1); [4] and [6] name weak effects as a likely explanation but do not test it against simple baselines.

Table K1. Positioning (see also App. A, Table 6).

PerturbationUnit of analysisA tie is read asHow to break it
[1] Systemageneticmetric; model × datasetmetric and data limit (systematic variation)a better metric
[5] Schäfer et al. (scPertEval)geneticmetricmetric limita better metric
[6] Heidari et al. (CrossSplit)chemical, geneticmetric; model × datasetmetric limit; data limit (hypothesis)a better metric
[2] PRESAGEgeneticmodel × datasetmetric and model limita better model
[3] scDrugPerturb-Benchchemicalmodel × datasetmetric and model limita better metric
[4] Lanxiang Li et al.genetic, chemicalmodel × datasetmetric and model limit; data limit (hypothesis)a better metric
Ourschemicalmodel × condition (drug × dose × cell line)metric, model or data limit: tested per conditionfix the model where there is room; otherwise a tie is expected

In short, the six works mostly look for a better metric or a better model, while we ask, for each condition, why the tie is there and whether it can be removed.

AuthorsOpenReview ↗

Response to Reviewer KSCd (2/3)

W2a / Q3. Do the conclusions hold on Tahoe-100M?

Yes. We re-ran the three diagnostics and the learning-free baselines on Tahoe-100M over five drug-held-out folds (379 drugs; Table K2, App. G.1). ChemCPA again fails to beat the compound-blind means (0.831 against 0.834 and 0.838), and each diagnostic gives the same verdict as on sci-Plex3:

  • Input use. The wrong compound changes ChemCPA's prediction by nothing (ΔR² 0.000 ± 0.000) and costs PerturbDiff 0.028 (sci-Plex3: −0.020 against −0.102): two models 0.034 R² apart differ qualitatively in whether the compound is used.
  • Response scaling. The highest slope is β = 0.188, so even the best model compresses the response to under a fifth of the observed rate; ChemCPA (0.001) and the baselines (≤ 0.005) are flat.
  • Stratified performance. PerturbDiff performs similarly to the per-dose mean in the weakest bin (0.968 against 0.967) and better in the strongest (0.447 against 0.326).

Values are lower because Tahoe-100M's compounds differ far less in effect strength (Q90/Q50 1.5–1.6 against 4.7–5.8; Table 21). We add two additional metrics (last two columns) to address W2c.

Table K2. Tahoe-100M, five folds (379 held-out drugs). Weakest and strongest of six effect-size bins: DEG50 R². ΔR² (T2) and DES recall apply to models only. Bold: best, and largest drop for ΔR² (T2).

MethodDEG50 R²ΔR² (T2)β (full)β (DEG50)Weakest binStrongest binPDSDES recall
PerturbDiff0.865 ± 0.025−0.028 ± 0.0130.188 ± 0.0610.178 ± 0.0740.968 ± 0.0020.447 ± 0.1400.825 ± 0.0150.249 ± 0.009
ChemCPA0.831 ± 0.0240.000 ± 0.0000.001 ± 0.0380.034 ± 0.0250.941 ± 0.0060.383 ± 0.1230.635 ± 0.0130.008 ± 0.000
Control mean0.794 ± 0.031–0.000 ± 0.0000.000 ± 0.0000.962 ± 0.0020.176 ± 0.1690.500 ± 0.000–
Perturbation mean0.834 ± 0.026–0.000 ± 0.0020.005 ± 0.0090.967 ± 0.0010.310 ± 0.1440.596 ± 0.011–
Per-dose perturbation mean0.838 ± 0.025–0.005 ± 0.0030.014 ± 0.0090.967 ± 0.0010.326 ± 0.1420.611 ± 0.011–

Full results. App. G: this table and the five-fold version of Table 4.

AuthorsOpenReview ↗

Response to Reviewer KSCd (3/3)

W2b / Q2. Cell count and the uncertainty of the effect magnitude

Thank you for raising this point. We agree that an effect size estimated from pseudo-bulk differences can be sensitive to sampling noise, and have revised the analysis; the stratified results hold (Table K3).

1. "Effect magnitude is estimated from pseudobulk differences and is therefore sensitive to sample noise."

We now estimate δ from two disjoint halves of each condition's cells (the split used for the technical duplicate of Miller et al., 2025), which removes the contribution of sampling noise; all bins use this estimate.

2. "A strong negative correlation between effect magnitude and cell count."

The strongest bin has the fewest cells (median 148, against 227 in the weakest), yet interestingly, its effect is the most reliably estimated: 3.0 times its sampling noise, against 0.15 in the weakest bin. This ratio divides the noise-corrected effect size of a condition by its sampling noise, the part of the raw effect size that the correction removes (noise² = raw² − corrected²); a value above 1 means the effect is larger than the noise. The strong effects are therefore robust signal and not an artefact of low cell numbers, whereas the weak bins are dominated by noise despite having more cells.

3. "Do the stratified results remain consistent?" (Q2)

Yes. With every condition subsampled to 50 cells and effect sizes, bins and scores recomputed, 94% of strongest-bin conditions stay in that bin, and PerturbDiff's position relative to the per-dose mean is unchanged in every bin (App. D.3, Tables 15 and 16).

What separates a data limit from a model limit

Per bin we also report an attainable-performance reference: one half of a condition's cells scored against the other, i.e. what a replicate measurement would score (per effect-size bin, unlike Schäfer et al., 2026, and Heidari et al., 2026). Its gap to the compound-blind per-dose mean (room-to-the-ceiling row, Table K3) is at most 0.016 DEG50 R² in the three weakest bins: a tie there is a limit of the data. In the strongest bin the gap is 1.08: a tie there is a limit of the model. For example, ChemCPA ties with the per-dose perturbation mean there, while PerturbDiff significantly outperforms it (Table J1 in our reply to Reviewer JgZs\color{purple}{\textbf{JgZs}}).

Table K3. sci-Plex3, five folds, per effect-size bin (1 = weakest). Last three rows: per-dose perturbation mean − PerturbDiff, DEG50 R² (negative: PerturbDiff is above). Bold: above in every fold.

Bin123456
Cells per condition (median)227230228225189148
Effect size / sampling noise (median)0.150.420.590.901.482.99
Room to the ceiling: ceiling minus per-dose mean+0.010+0.014+0.016+0.030+0.110+1.080
Noise-corrected bins, all cells+0.075+0.076+0.058+0.062+0.033−0.378
Noise-corrected bins, 50 cells per condition+0.067+0.070+0.064+0.058+0.034−0.387
Raw effect-size bins, all cells+0.072+0.068+0.061+0.067+0.035−0.379

Full results. Values above are means over the five folds; the standard deviations are in Table 14 (App. D, ceiling) and Table 15 (App. D.3, stratified results), which also give effect size from 10, 50, 200 and all cells and the per-bin scores of every method.

W2c. Biological correctness of the predicted gene-level responses

Thank you for this comment. By "mechanistic" we refer to a systematic study of how the model, the metric and the data interact, rather than to the biological meaning of the predicted responses; we have clarified this in the manuscript.

The point is an interesting one nonetheless, so we add two metrics from the Virtual Cell Challenge (Roohani et al., 2025): DES recall, whether a model recovers each condition's significant genes, and PDS, whether a predicted change is closer to its own condition's observed change than to other conditions'. Both agree with our diagnostics: PerturbDiff, whose predictions depend most on the compound, is highest (DES recall 0.141 ± 0.021, PDS 0.605 ± 0.039) and CondOT, which does not respond to the compound, lowest (Table R1, general response); the same ordering holds on Tahoe-100M (Table K2, part 2/3).

We hope these changes address your concerns, and we are glad to provide any further clarification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all the reviewers for their valuable and insightful feedback. We are glad that the reviewers found the question we ask interesting (JgZs\color{purple}{\textbf{JgZs}}), the analysis interpretable and informative beyond aggregate metrics (KSCd\color{blue}{\textbf{KSCd}}, EHut\color{teal}{\textbf{EHut}}), the input-usage diagnostic simple and tied to why these models can fail to outperform simple baselines (EHut\color{teal}{\textbf{EHut}}), and the condition-balanced sampling a simple fix that shows how relatively simple training modifications can improve perturbation prediction (KSCd\color{blue}{\textbf{KSCd}}, JgZs\color{purple}{\textbf{JgZs}}, EHut\color{teal}{\textbf{EHut}}), with the tools useful (JgZs\color{purple}{\textbf{JgZs}}).

In short. Everyone observes the tie with simple baselines. We ask why, per condition: for most conditions the data leave almost no room and a tie is expected; where effects are strong, some models do not use the compound, and for ChemCPA this can be fixed.

We found the constructive feedback of the reviewers very helpful and have prepared an updated manuscript. We have mainly updated the positioning in Section 1, the effect-size definition in Section 3.2, the results in Sections 5.3 and 5.4, Tables 1, 2, 3 and 5, Figure 2, and Appendices A, B.2, B.3, C.3, C.4, D, D.3, E and G; changes are marked in red. We reply to each reviewer in more detail in individual responses. We summarize the changes below:

  • We now report five folds for every method and reference, with mean ± standard deviation over folds, in two consolidated tables (Tables 1 and 5); the ordering on the diagnostics is stable across folds. (Reviewers JgZs\color{purple}{\textbf{JgZs}}, EHut\color{teal}{\textbf{EHut}})
  • We have added PDS, DES recall and a BH-FDR gene set for every method, and a controlled comparison showing what β adds to R² (Appendix C.3). (Reviewers EHut\color{teal}{\textbf{EHut}}, KSCd\color{blue}{\textbf{KSCd}})
  • We have added a per-effect-size-bin reproducibility ceiling and a per-cell-line, per-dose reference, against which every margin is now measured, and redrawn Figure 2 as paired differences to that reference (Appendix D). (Reviewers JgZs\color{purple}{\textbf{JgZs}}, KSCd\color{blue}{\textbf{KSCd}})
  • We now estimate effect size from two disjoint half-samples and repeat the stratified analysis at equal cell number; the stratified results hold (Appendix D.3). (Reviewers KSCd\color{blue}{\textbf{KSCd}}, JgZs\color{purple}{\textbf{JgZs}})
  • We have run the three diagnostics on Tahoe-100M over five folds (379 held-out drugs); every conclusion from sci-Plex3 holds (Appendix G). (Reviewers KSCd\color{blue}{\textbf{KSCd}}, JgZs\color{purple}{\textbf{JgZs}})
  • We have positioned our work against the six related works supplied by Reviewer KSCd\color{blue}{\textbf{KSCd}} in a new table (Appendix A, Table 6) and clarified why every result is read per condition: a compound can do nothing at one dose and a great deal at another. (Reviewers KSCd\color{blue}{\textbf{KSCd}}, EHut\color{teal}{\textbf{EHut}})

Table R1. sci-Plex3, five folds, all methods. Mean ± standard deviation over folds. ΔR² (T2): change in DEG50 R² when the compound embedding is swapped with another drug's. β: slope of predicted on observed effect size. Bold: best. Doloris: three folds (Appendix B.2).

MethodDEG50 R²ΔR² (T2)βPDSDES recall
CondOT0.768 ± 0.0690.000 ± 0.0000.048 ± 0.0120.529 ± 0.0040.001 ± 0.000
Doloris (3 folds)0.769 ± 0.037−0.035 ± 0.0150.241 ± 0.0250.572 ± 0.0010.056 ± 0.011
PerturbDiff0.789 ± 0.070−0.102 ± 0.0330.408 ± 0.2100.605 ± 0.0390.141 ± 0.021
ChemCPA0.732 ± 0.075−0.020 ± 0.0190.104 ± 0.0840.541 ± 0.0110.029 ± 0.003
ChemCPA, condition-balanced0.786 ± 0.080−0.041 ± 0.0220.153 ± 0.1030.540 ± 0.0170.028 ± 0.002
ChemCPA, cond.-bal. + dir. loss0.747 ± 0.061−0.075 ± 0.0460.224 ± 0.1420.540 ± 0.0280.038 ± 0.006
Control mean0.743 ± 0.082–00.500–
Perturbation mean0.768 ± 0.077–−0.001 ± 0.0010.526 ± 0.003–
Per-dose perturbation mean0.776 ± 0.075–0.044 ± 0.0090.534 ± 0.005–

We appreciate the reviewers' time and are happy to provide any further analysis or clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.