- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper benchmarks three transport-based models (CondOT, Doloris, and PerturbDiff) against a latent-disentanglement baseline (ChemCPA) and simple linear baselines on two chemical perturbation datasets, evaluating generalization to unseen drugs. It proposes an inference-time diagnostic that quantify how much a model's predictions depend on the perturbation identity (input usage). The paper also evaluates response scaling, defined as the slope of an OLS regression of predicted on true per-gene expression changes, together with coeficient of determination () stratified by effect size. It claims that models that make use of perturbation information, notably PerturbDiff, achieve competitive response scaling and improved on high-effect perturbations relative to the linear baselines. Finally, an ablation on ChemCPA that adds conditional sampling and a perturbation-induced signal is reported to improve it on all three axes: input usage, response scaling, and effect-size-stratified performance.
Critical strengths
- The paper benchmarks deep learning models and introduces a simple inference-time diagnostic for perturbation input usage. It ties low input usage to why these models can fail to outperform simple baselines.
- The paper proposes a fix for ChemCPA that improves perturbation usage and shows gains on all three axes: input usage, response scaling, and effect-size-stratified performance.
Critical weaknesses
Novelty and contribution The paper claims to improve on several previously proposed benchmarks by focusing on chemical perturbations. The approach and conclusions appear to be a straightforward extension of those benchmarks, with conclusions consistent with those for genetic perturbations. Simple baselines remain competitive, even on the chemical perturbation prediction task, which limits the significance of the findings.
Motivation and redundancy of the proposed diagnostics
- The proposed diagnostics (input usage, the coefficient, and stratification by effect size) can largely be captured by existing metrics. Input usage: it is unclear why the perturbation discrimination score (PDS) is not used, since it could capture input usage more cheaply. coefficient: the motivation is unclear, since is already highly correlated with . It is also uninformative for noisy predictions, which can still attain a high .
Experimental design and evaluation
- Cross-validation: the main paper appears to report a single fold of the 80/20 split. It is unclear why the 5-fold results are not consolidated and reported in the main text.
- DEG selection: the fixed top-50 DEG selection could be strengthened by using a Benjamini–Hochberg FDR threshold, since some perturbations may have fewer than 50 significant DEGs.
- Metrics: it is unclear why DES recall is not included in the evaluation.
Baselines and completeness The paper proposes cell-level linear baselines (scControl and scPerturb) but never benchmarks against them. Not all tables and results include all linear baselines and deep learning models, which makes comparisons inconsistent.
Reporting and statistical rigor
- Results are spread across several tables. A single table for the 5-fold setup, including all metrics (with , input usage, and the stratified results), would be clearer.
- The paper should consistently report the mean and standard deviation across all metrics. These appear to overlap substantially between the best-performing deep learning models and the linear baselines, which weakens the claimed differences.
- Figure 2: the error bars overlap substantially, making the figure difficult to interpret. Effect strength could be reported over quantiles (instead of 6 bins) as numerical values.
ChemCPA modification and ablation
- The ChemCPA setup needs further explanation. It is unclear what the conditional sampling approach is and how the auxiliary perturbation loss is included.
- Table 5: the ChemCPA results are mixed. For instance, higher input usage does not translate into higher . The gains appear only on the proposed response scaling metric and on high-effect perturbations. This is somewhat circular, since the improvement shows up mainly on the metrics the paper itself introduces, and it weakens the claim of improvement on all three axes.
Questions for authors
- Could you report the perturbation discrimination score (PDS) for all models and compare it with the proposed input-usage diagnostic?
- Could you clarify what the coeffienct metrics adds beyond ?
- Could you extend the ChemCPA results (Table 5) to include all linear and deep learning baselines under the same set of metrics, so that the effect of the modification can be compared directly with the other methods? Also, could you explain why, and show whether the improvements hold on metrics other than the proposed ones (PDS, DES recall)?
- Could you describe the conditional sampling approach and how the auxiliary perturbation loss is incorporated?
- What conclusions here are specific to chemical perturbations, as opposed to extending those already found for genetic perturbations?
8 comments
Response to Reviewer EHut (2/3)
W2 / Q1 / Q2. Are the diagnostics captured by existing metrics?
1. Input Use and PDS (Q1).
Thank you for the suggestion; PDS is now reported for every model (Table R1; chance 0.5). PDS compares predictions with observations, so a low score does not say whether the model ignored the compound or used it and got it wrong. Input Use intervenes on the compound input of the trained model at inference time, which separates those two cases. The two can disagree: on Tahoe-100M, ChemCPA reaches a PDS of 0.635, above the compound-blind per-dose mean (0.611), yet swapping the compound changes its prediction by 0.000 (Table K2 in our reply to Reviewer ).
2. What β adds to R² (Q2).
We thank the reviewer for the interesting question. There is a real link in general, and it is worth stating precisely: for a single regression of on , , so the slope and the of that regression carry the same information up to a variance ratio. If the we report were the of β's own regression, β would indeed be redundant. It is not. The two are computed on different data points: for R², a data point is a gene; for β, it is a condition:
| One data point | Computed over | |
|---|---|---|
| R² | a gene | the genes of one condition (full panel or DEG50), then averaged over conditions |
| β | a condition | the held-out conditions |
Because the two run over different things, one can be high while the other is zero: a predictor that reproduces the baseline and none of the response still explains most of the variance in expression levels, yet has β exactly 0. That is the control mean (Table E2). β is also a slope rather than a correlation, so it registers systematic shrinkage: a prediction can order conditions perfectly and still sit well below β = 1, which is the second row of Table E2.
Table E2. R² and β measure different things (sci-Plex3, five folds). Controlled predictors, not models.
| Predictor | DEG50 R² | β |
|---|---|---|
| Control mean (no response) | 0.743 ± 0.082 | 0.00 |
| True response × 0.3 (shrunk) | 0.874 ± 0.040 | 0.30 |
3. "β is uninformative for noisy predictions."
We agree that β alone can stay high for a noisy prediction as it measures the size of the response, not its direction or which genes move. That is why we report it alongside DES recall and PDS. On this dataset the three agree about which models use the compound: PerturbDiff is highest on all three and CondOT lowest (Table R1).
Full results. App. C.3: the controlled predictors with DEG50 and full-panel R² and β.
W3. Single fold; top-50 vs BH-FDR; DES recall
Thank you for these suggestions; all three are implemented (Table R1 in the general response; revised Table 1).
1. "A single fold of the 80/20 split."
Every method is now reported on five folds in the main text, except Doloris, which ran on three: on folds 1 and 3 its mask model collapsed with each of three random seeds, and recovering them would have required fold-specific tuning that no other method received (App. B.2).
2. "A Benjamini–Hochberg FDR threshold."
Thank you for the suggestion. We followed ChemCPA to obtain the DEG50 for the sci-Plex3 dataset and STATE for the Tahoe-100M dataset. However, we agree this is an interesting point, so we added FDR-set , computed on each condition's significant genes at BH-FDR 5%. We observe that only 723 of 1,788 conditions have ten or more significant genes, and the median is one to two in the three weakest effect-size bins, so for most conditions a fixed top-50 panel is larger than the set of genes that actually move. The conclusions are unchanged on the FDR set: PerturbDiff 0.56 against 0.38 for the per-dose perturbation mean.
3. "DES recall is not included."
Added for every method in Table 1; PerturbDiff is the highest in every fold.
W4. scControl / scPerturb; incomplete tables
Thank you for pointing this out, we have clarified the notation of the method: scControl and scPerturb were benchmarked, but their values sat in Table 1's single-cell columns under the pseudo-bulk row names. Table 1 now labels both, and revised Tables 1 and 5 list every model and every reference on sci-Plex3.
Response to Reviewer EHut (3/3)
W5. Scattered tables; overlapping ±; Fig. 2; quantiles
Thank you for these suggestions on reporting; we have followed each. Two five-fold tables now hold everything: Table 1 the accuracy metrics, Table 5 the three diagnostics together with the stratified results.
1. "Mean and standard deviation overlap between models and baselines", and "Fig. 2: the error bars overlap."
The revision reports mean ± standard deviation over folds throughout, and Fig. 2 now plots each method's paired difference to the per-dose perturbation mean. We do not claim a difference in aggregate accuracy between the models. The aim is to explain why they score so close to the learning-free baselines, and the diagnostics of Table 5, where the methods do differ, are what address that.
2. "Effect strength over quantiles, as numerical values."
The six bins are already equal-count quantiles of effect size, and App. D reports every per-bin value numerically.
W6 / Q3 / Q4. The ChemCPA variants
1. "What is the conditional sampling approach, and how is the auxiliary loss included?" (Q4)
Thanks for your question. The description is now in App. B.3. Condition-balanced sampling draws 64 training conditions uniformly per batch, then 32 cells within each. The directional loss adds cosine and magnitude terms on per-condition residuals to the per-dose training mean. Both settings were selected on a validation partition built from the training drugs only.
2. "The gains appear only on the metrics the paper introduces." (Q3)
Thank you for the question. Condition-balanced sampling improves three metrics we did not introduce: DEG50 R² 0.732 to 0.786, full-panel R² 0.884 to 0.922, FDR-set R² 0.291 to 0.380 (Table E3). The directional loss also improves DES recall, which you suggested, from 0.029 to 0.038.
3. "Higher input usage does not translate into higher R²", and improvement on all three axes.
Thank you for the question! The sampler improves accuracy and compound use together, while the directional loss raises compound use further (Input Use −0.041 to −0.075, β 0.15 to 0.22) at a small cost in aggregate accuracy (full-panel R² 0.865 against 0.884). The aggregate averages two regimes: in the weak bins the ceiling leaves at most 0.016 above the per-dose perturbation mean, so a model that uses the compound has nothing to win there and pays a little instead, while in the strongest bin both variants move from 0.012 below that reference to about 0.115 above it. PerturbDiff shows the same pattern.
Table E3. ChemCPA and its variants (five-fold means). Strongest bin: DEG50 R² minus that of the per-dose perturbation mean, in the last of six effect-size bins.
| ChemCPA | DEG50 R² | Full R² | FDR-set R² | DES recall | ΔR² (T2) | β (full) | Strongest bin |
|---|---|---|---|---|---|---|---|
| Original | 0.732 | 0.884 | 0.291 | 0.029 | −0.020 | 0.104 | −0.012 |
| + condition-balanced sampling | 0.786 | 0.922 | 0.380 | 0.028 | −0.041 | 0.153 | +0.115 |
| + sampling + directional loss | 0.747 | 0.865 | 0.288 | 0.038 | −0.075 | 0.224 | +0.116 |
4. "Extend Table 5 to all baselines under the same metrics." (Q3)
Thank you for your suggestion! We have revised Tables 1 and 5 to include both variants.
We hope these changes address your concerns. If any point remains open, please tell us and we will answer it.
Response to Reviewer EHut (1/3)
We thank the reviewer for the thorough review, which led to a more complete and more consistent evaluation. We summarize the major changes first and respond to each of your questions in more detail after this:
1. "A straightforward extension of those benchmarks."
We agree that this is an active field and that a tie with simple baselines is a common observation; our work is indeed inspired by those benchmarks. As our positioning table shows (Table K1 in our reply to Reviewer KSCd), our main contribution is to understand why the tie happens, what it indicates, and how it can be fixed. Existing benchmarks mostly propose a better metric or a better model. Only Systema (Viñas Torné et al., 2025) studies the cause systematically from the data perspective, and it does so for genetic perturbations. Chemical perturbation data have a different nature: a compound is applied across doses and may do nothing at one dose and a great deal at another (Srivatsan et al., 2020), so the tie has to be resolved per condition: where the data leave no room it is expected, and where they do it is a limit of the model. None of the six works substitutes the model's own perturbation input or regresses predicted magnitude on observed. This is also our answer to Q5.
2. "Simple baselines remain competitive, which limits the significance."
That is our starting point, not our main finding. Where effects are weak little is available to win: 60% of held-out conditions have fewer than 10 significant genes (BH-FDR 5%), and a split-half reproducibility ceiling (the technical duplicate of Miller et al., 2025) sits at most 0.016 DEG50 R² above the per-dose perturbation mean. Where they are strong the gap belongs to the model: PerturbDiff exceeds the per-dose mean by 0.378 in the strongest bin, while models with similar aggregate DEG50 R² depend far less on the compound (Table R1). And it is actionable: ChemCPA's sampler alone raises DEG50 R² from 0.732 to 0.786.