- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PIE predicts population-level cellular perturbation effects rather than individual post-perturbation cells. It combines biological knowledge embeddings, control expression, and summaries of observed perturbation responses through a Perceiver IO architecture. Three output heads predict differential expression (DE), log fold changes, and discretized expression changes. The model supports unseen contexts, unseen perturbations, their combination, and cross-dataset transfer. However, the last comparison is against a zero-shift control baseline, making it substantially weaker.
Critical strengths
-
The paper directly predicts population-level effects aligns training with common downstream analyses. This also avoids requiring individual-cell counterfactual targets that are not directly observed.
-
The paper evaluates across contexts, perturbations, and datasets. Its joint generalization is particularly relevant to experimental prioritization.
-
Training all subsets of input groups and computing exact Shapley values provides useful evidence about which signals matter. The authors also acknowledge results that weaken their narrative.
Critical weaknesses
-
The headline AUPRC improvements do not clearly establish perturbation-specific generalization. This is my main concern. Section 4.4 reports that, when both entities are unseen, adding all five input groups improves AUPRC by only 0.010. Combined with the reported full-model result, the model without these groups achieves approximately 0.229 versus 0.239. Under the described architecture, that ablated model retains output-gene embeddings but receives no condition-specific biological inputs. Its performance therefore appears largely attributable to a learned, broadly shared ranking of genes likely to become DE. Similarly, for unseen perturbations, the total input-group contribution is only 0.045—much smaller than the headline improvement over competing methods.
-
In the jointly unseen setting, comparing only against zero expression change is insufficient. Simple models can accept unseen context and perturbation embeddings—for example, ridge regression, a bilinear model, or an MLP trained on the same targets and inputs. TxPert (https://www.nature.com/articles/s41587-026-03113-4) is also a particularly relevant missing experimental baseline. It is cited in the submission and evaluates both unseen genetic perturbations and cross-cell-line transfer using biological knowledge graphs. Comparisons in compatible settings would strengthen the state-of-the-art claim.
-
Population-level summary-statistic prediction already appears in Rhaister (https://www.biorxiv.org/content/10.64898/2026.06.09.731197v1.full.pdf), which directly models fold changes, significance statistics, and expression deltas. Thus, this formulation alone is not new. PIE’s contribution is better understood as combining that formulation with richer knowledge inputs and a flexible architecture. However, the current ablations primarily isolate input sources. They do not establish how much improvement requires.
-
Some auxiliary inputs, including DepMap and L1000, contain independently measured perturbational information. This is legitimate, but differs from predicting a perturbation with no prior experimental characterization. Training time, parameter count, inference cost, and embedding-preparation costs would help assess whether the architecture offers practical advantages over simpler alternatives.
Questions for authors
-
How much performance is perturbation-specific? Can you report gene-wise DE-frequency and gene-only baselines, together with perturbation-input shuffling? In particular, how do you interpret the approximately 0.229 AUPRC without the five input groups versus 0.239 with them when both entities are unseen?
-
How robust are the gains to output conversion? Do the AUPRC improvements persist with a shared DE-calibration procedure for pseudobulk models, different synthetic-cell counts, and reduced p-value saturation?
11 comments
Question 1: How much performance is perturbation-specific
Continuing from previous comment on Question 1: How much performance is perturbation-specific
Replogle-Nadig within dataset, joint unseen
| Model | AUPRC | Jaccard score | DE direction match | Spearman Effect Size | Spearman LFC | Discrimination Score L1 | Pearson delta |
|---|---|---|---|---|---|---|---|
| PIE | 0.239 ± 0.054 | 0.116 ± 0.022 | 0.721 ± 0.022 | 0.450 ± 0.061 | 0.460 ± 0.059 | 0.643 ± 0.049 | 0.231 ± 0.031 |
| Context mean | 0.088 ± 0.018 | 0.091 ± 0.041 | 0.000 ± 0.000 | undefined | undefined | 0.502 ± 0.000 | undefined |
| Population statistics, global | 0.230 ± 0.060 | 0.011 ± 0.015 | 0.668 ± 0.018 | undefined | 0.362 ± 0.113 | 0.502 ± 0.000 | 0.263 ± 0.089 |
| Gene-text MLP | 0.240 ± 0.055 | 0.116 ± 0.034 | 0.641 ± 0.038 | undefined | 0.330 ± 0.092 | 0.503 ± 0.001 | 0.221 ± 0.075 |
| Gene-only PIE | 0.233 ± 0.050 | 0.117 ± 0.034 | 0.672 ± 0.025 | undefined | 0.354 ± 0.109 | 0.503 ± 0.001 | 0.218 ± 0.071 |
PIE outperforms these population baselines on unseen context generalization. For unseen perturbation generalization, PIE outperforms baselines on all metrics except Pearson Delta. The joint unseen setting is the hardest. PIE is at par with these population baselines on AUPRC and Jaccard score, while outperforming on other DE-focused metrics like DE direction match and Spearman LFC. This shows that PIE is able to predict perturbation specific direction and magnitude for the joint unseen setting. PIE also outperforms these population baselines on Discrimination score L1, which requires perturbation specific predictions to perform better than random.
Shuffling perturbation specific inputs during inference: Starting with the trained PIE checkpoint on the within dataset task for Replogle-Nadig, we shuffle the knowledge sources, the same perturbation evidence or both at inference time. In the tables below, we compare these ablations to PIE (no shuffle) across all three generalization settings.
Replogle-Nadig within dataset, unseen context, seen perturbation
| Model | AUPRC | Jaccard score | DE direction match | Spearman effect size | Spearman LFC | Discrimination Score L1 | Pearson delta |
|---|---|---|---|---|---|---|---|
| PIE (no shuffle) | 0.424 ± 0.034 | 0.185 ± 0.023 | 0.819 ± 0.026 | 0.734 ± 0.052 | 0.664 ± 0.046 | 0.764 ± 0.061 | 0.313 ± 0.026 |
| Knowledge sources shuffled | 0.292 ± 0.043 | 0.093 ± 0.014 | 0.718 ± 0.030 | 0.138 ± 0.032 | 0.472 ± 0.089 | 0.550 ± 0.011 | 0.203 ± 0.052 |
| Same-pert evidence shuffled | 0.258 ± 0.051 | 0.131 ± 0.017 | 0.772 ± 0.029 | 0.689 ± 0.054 | 0.550 ± 0.044 | 0.716 ± 0.068 | 0.269 ± 0.030 |
| Both shuffled | 0.186 ± 0.055 | 0.066 ± 0.016 | 0.612 ± 0.034 | 0.005 ± 0.033 | 0.239 ± 0.101 | 0.500 ± 0.003 | 0.125 ± 0.041 |
Replogle-Nadig within dataset, seen context, unseen perturbation
| Model | AUPRC | Jaccard score | DE direction match | Spearman effect size | Spearman LFC | Discrimination Score L1 | Pearson delta |
|---|---|---|---|---|---|---|---|
| PIE (no shuffle) | 0.304 ± 0.023 | 0.168 ± 0.011 | 0.775 ± 0.022 | 0.525 ± 0.030 | 0.552 ± 0.022 | 0.709 ± 0.006 | 0.339 ± 0.023 |
| Knowledge sources shuffled | 0.213 ± 0.013 | 0.093 ± 0.006 | 0.646 ± 0.014 | −0.005 ± 0.016 | 0.308 ± 0.021 | 0.498 ± 0.007 | 0.194 ± 0.022 |
Replogle-Nadig within dataset, joint unseen
| Model | AUPRC | Jaccard score | DE direction match | Spearman effect size | Spearman LFC | Discrimination Score L1 | Pearson delta |
|---|---|---|---|---|---|---|---|
| PIE (no shuffle) | 0.239 ± 0.054 | 0.116 ± 0.022 | 0.721 ± 0.022 | 0.450 ± 0.061 | 0.460 ± 0.059 | 0.643 ± 0.049 | 0.231 ± 0.031 |
| Knowledge sources shuffled | 0.191 ± 0.055 | 0.072 ± 0.022 | 0.630 ± 0.031 | −0.020 ± 0.052 | 0.271 ± 0.096 | 0.499 ± 0.007 | 0.143 ± 0.048 |
Upon shuffling the perturbation-specific input components at inference, we observe that the model underperforms on all metrics across all three generalization settings. For all three generalization settings, removing all sources of perturbation signal reduces the Spearman effect size and Discrimination score L1 metrics to those equivalent to a random predictor.
These results suggest PIE is able to leverage its perturbation signal carrying inputs to predict perturbation-specific responses, even for unseen perturbations.
Question 1: How much performance is perturbation-specific
Continuing from previous comment on Question 1: How much performance is perturbation-specific
Interpreting the contribution of perturbation knowledge sources: We also acknowledge the reviewer observation regarding pointing out the small contribution of input components towards AUPRC in the joint unseen setting. During the revision process, we realized that our attribution analysis, as shown in Figure 4, was based on an older formulation of PIE. We have re-run our attribution analysis on the canonical version that is reflected in tables 3–5, and have updated Figure 4. Based on these attribution analyses, in the joint unseen setting, the gain in AUPRC between all input components on and off is +0.006.
As shown in the results above, the joint unseen setting is the hardest generalization setting. In this setting, the knowledge sources are the only perturbation-specific input to PIE. In this setting, PIE with knowledge sources available performs at par with population level baselines when predicting DE membership, as shown by AUPRC and Jaccard score metrics. Training without knowledge sources leads to a comparable checkpoint. However, in this generalization setting, PIE is able to leverage the knowledge sources to gain an advantage in predicting DE direction and magnitude, as well as predicting a discriminative pseudobulk delta expression. This is also reflected in the Shapley attribution scores in this setting, with knowledge sources having a negative contribution of −0.012 and −0.006 towards AUPRC and Jaccard score respectively, while having positive contributions of +0.060, +0.069 and +0.141 towards DE direction match, Spearman LFC and Discrimination Score L1 respectively. This further highlights that the different input components of PIE, including perturbation knowledge sources, vary in contribution across generalization settings and metrics.
Question 1: How much performance is perturbation-specific
We appreciate the reviewer’s suggestion to probe the perturbation specific response prediction of PIE. For this investigation, we developed four new baselines. Across all but a few metrics, PIE consistently outperforms all baselines. Moreover, we perform an ablation where we shuffle the perturbation specific input, namely the knowledge sources and same perturbation evidence, across perturbations during inference. PIE’s performance in all generalization settings, including joint unseen setting, degrades when the perturbation specific input is shuffled. These results show that PIE’s predictions are perturbation specific, even in the unseen perturbation and joint unseen settings.
Population summary baselines and results:
We define the baselines as below:
In the tables below, we compare these baselines and Context Mean (Adduri et al., Cell 2026) to PIE. These results obtained on the within dataset task on Replogle-Nadig (Replogle et al., Cell 2022; Nadig et al., Nature Genetics 2025), where we train each model on the train set, and evaluate on all three generalization settings.
Replogle-Nadig within dataset, unseen context, seen perturbation
Replogle-Nadig within dataset, seen context, unseen perturbation