- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PROVE improves cross-context single cell perturbation predictions by modifying the sampling direction of a pre-trained model (no retraining is required at all). At each generation step it predicts the final cell and then compares that to reference statistics derived from the training data: the mean response and the variation between context-level mean responses for the same perturbation in other contexts; the responses to other perturbations in the target context. These comparisons flag three possible inconsistencies with the references: an effect that is too small (effect collapse), a prediction more aligned with another perturbation’s response direction than with the target reference direction (perturbation confusion), and a response that is very different from what we’d expect from other contexts (context drift). PROVE generates a correction direction for each of these inconsistencies and then combines those with the model’s own direction using a minimum norm (MGDA style) consensus. It then steers the velocity towards the resulting point whilst maintaining its magnitude. Experiments cover three flow models (scDFM, CellFlow, CFM-GP) on three datasets (Replogle, Frangieh21, Srivatsan20), plus additional experiments on two diffusion models, ablations, pathway analysis and comparisons against the reference-only predictors.
Critical strengths
-
Consistent improvements as a plug-in. PROVE requires no retraining. Table 1 shows gains on all five perturbation-specific metrics across all nine model/dataset combinations. Results over three seeds (with standard deviations) are in the appendix and the diffusion model and pathway analyses provide further support.
-
Transparent comparisons and limits. Tables 6-8 make a direct comparison of PROVE with both the reference transfer and the regression baselines (even in cases where the comparison is not in PROVE’s favour). The authors also split agreement with the reference patterns from actual biological accuracy (Section 4.2.2, Appendix C.2) which makes the evidence much easier to assess.
Critical weaknesses
-
The benefit of the full editing procedure over simpler alternatives is not fully established. The ablation study does compare minimum norm consensus with normalized arithmetic averaging, but only for CellFlow on Replogle. The ablation doesn’t include comparisons with endpoint only correction or with direct interpolation between the backbone predictions and the reference responses (as acknowledged on p9 lines 447–449). These would have helped establish whether editing at each sampling step was required.
-
Table 5 shows that the effect collapse check activates on the ground-truth mean responses of 25.6% of the held-out context–perturbation pairs in Srivatsan20. The paper admits that performance is not assessed separately on those cases. The stats on variation between context means are also applied to the individual generated cells. The aggregate improvements therefore don’t tell us whether the guidance is in fact harming valid responses that differ from the references or whether it’s actually reducing the meaningful variation from cell to cell.
Questions for authors
-
Does PROVE perform better than correcting only the final prediction or taking a weighted average of the model’s prediction and a reference-based prediction, when all three approaches use the same reference data and comparable parameter tuning?
-
Does PROVE improve or degrade the predictions on the target pairs whose ground truth mean responses are what cause the reference checks to be triggered (i.e. the 25.6% that trigger the effect collapse check in Srivatsan20)? What's the effect on within condition variability?
-
Do the component ablation results generalise beyond CellFlow on Replogle? What's the actual measured inference time overhead relative to the original sampler?
-
The redirection preserves the magnitude of velocity and Appendix A.7 states that the correction is zero when the consensus direction is equal to the backbone direction. How does PROVE deal with effect collapse in this aligned case where the backbone is pointing in the right direction but the effect is too weak?
-
Even though you are following the official protocol, why do the vanilla scDFM (Frangieh21, Srivatsan20) and the retrained Squidiff (Replogle) give negative or near zero correlations? Can you confirm that the preprocessing is compatible and that they do in fact give the expected baseline performance?
9 comments
Q2. Performance on Ground-truth Prior-violating Conditions and Preservation of Within-condition Variability
(a) Performance on ground-truth prior-violating conditions
To address the concern that reference priors may incorrectly modify genuine context-specific biological responses, we conducted a ground-truth prior-violation stratification analysis.
Specifically, we identified target conditions whose ground-truth responses violate at least one reference prior (EC, PC, or CD) and directly evaluated PROVE's predictive performance on these conditions. Ground-truth responses were used exclusively for post-hoc stratification and evaluation, and were not involved in inference or reference construction.
The results are reported below ().
| Dataset | Δ Pearson-Δ | Δ DE-Spearman | Δ DS | Δ AUPRC-DE |
|---|---|---|---|---|
| Replogle | +0.0378 | +0.1427 | +0.0682 | +0.0739 |
| Frangieh21 | +0.0906 | +0.4120 | +0.0584 | +0.5058 |
| Srivatsan20 | -0.0112 | +0.0817 | +0.0206 | +0.0202 |
Even when ground-truth responses violate the reference priors, PROVE achieves positive improvements in 11 out of 12 dataset–metric comparisons.
These results indicate that disagreement between genuine biological responses and reference priors does not necessarily cause PROVE to suppress valid context-specific biological effects. Even under these more challenging conditions, PROVE continues to improve most perturbation-specific prediction metrics.
We note that this analysis uses the broader any-prior violation definition, which includes the EC-violating conditions highlighted by the reviewer, rather than separately reporting results on only the 25.6% EC-triggered conditions in Srivatsan20.
(b) Preservation of within-condition variability
To evaluate whether PROVE compromises genuine within-condition variability, we assessed three gene-wise variance consistency metrics:
- Variance Spearman: Rank correlation between predicted and ground-truth gene-wise variances.
- Top-20% Jaccard: Overlap between the predicted and ground-truth sets of the top 20% most variable genes.
- NDCG@20%: How accurately highly variable genes are ranked near the top.
The changes relative to Vanilla are reported below.
| Dataset | Δ Variance Spearman | Δ Top-20% Jaccard | Δ NDCG@20% |
|---|---|---|---|
| Replogle | +0.0005 | +0.0080 | +0.0097 |
| Frangieh21 | -0.0040 | +0.0080 | +0.0026 |
| Srivatsan20 | +0.0012 | +0.0011 | -0.0009 |
Overall, PROVE maintains gene-wise variability structure comparable to that of the vanilla backbone across all three datasets, without introducing substantial or systematic degradation.
Q3. Generalization of Component Ablations and Measured Inference-time Overhead
(a) Component ablation
The component ablation study in the original submission was primarily conducted using CellFlow on Replogle. To examine whether these observations generalize to another dataset, we performed leave-one-component-out ablations using CellFlow on Srivatsan20.
The results are presented below.
| Method | MAE↓ | Pearson-Δ↑ | DE-Spearman↑ | DS↑ | MMD-GEX↓ | MMD-PCA↓ | AUPRC-DE↑ | DEG Recall↑ |
|---|---|---|---|---|---|---|---|---|
| Full PROVE (Table 1) | 0.0128 | 0.8772 | 0.8754 | 0.8597 | 0.0100 | 0.1251 | 0.6690 | 0.7523 |
| w/o EC | 0.0135 | 0.8738 | 0.8750 | 0.8451 | 0.0120 | 0.1252 | 0.6600 | 0.7488 |
| w/o PC | 0.0130 | 0.8756 | 0.8545 | 0.8570 | 0.0117 | 0.1339 | 0.6518 | 0.7086 |
| w/o CD | 0.0141 | 0.8717 | 0.8723 | 0.8603 | 0.0109 | 0.1301 | 0.6654 | 0.7523 |
| w/o Pareto | 0.0138 | 0.8757 | 0.8738 | 0.8606 | 0.0122 | 0.1252 | 0.6672 | 0.7520 |
| w/o final norm | 0.0137 | 0.8770 | 0.8754 | 0.8595 | 0.0120 | 0.1251 | 0.6690 | 0.7523 |
These results support the effectiveness of Full PROVE in this setting. In particular, removing either EC or PC degrades all eight metrics, supporting the role of these reference constraints in recovering perturbation responses.
Removing the other components also fails to yield consistent improvements, consistent with their complementary roles within the complete algorithm.
This experiment extends the original ablation observations from Replogle to Srivatsan20 using CellFlow. Component ablations on the other backbones are ongoing and will be provided subsequently.
(b) Measured inference-time overhead
We measured the mean per-cell wall-clock inference time across three backbones and three datasets, covering all nine combinations. For each combination, latency was averaged over the same 1,000 source cells and three paired repetitions. The table below reports mean per-cell sampling time.
| Backbone | Dataset | Vanilla (s/cell) | PROVE (s/cell) | Extra time (s/cell) | Ratio |
|---|---|---|---|---|---|
| scDFM | Replogle | 3.918 | 4.432 | +0.515 | 1.13× |
| scDFM | Frangieh21 | 3.867 | 4.507 | +0.639 | 1.17× |
| scDFM | Srivatsan20 | 3.879 | 4.538 | +0.659 | 1.17× |
| CellFlow | Replogle | 0.033 | 0.472 | +0.439 | 14.22× |
| CellFlow | Frangieh21 | 0.035 | 0.304 | +0.268 | 8.63× |
| CellFlow | Srivatsan20 | 0.034 | 0.202 | +0.168 | 5.95× |
| CFM-GP | Replogle | 0.0028 | 0.363 | +0.361 | 130.99× |
| CFM-GP | Frangieh21 | 0.0028 | 0.321 | +0.319 | 115.06× |
| CFM-GP | Srivatsan20 | 0.0029 | 0.232 | +0.229 | 80.28× |
The results show that the relative inference-time overhead is 1.13–1.17× for scDFM, 5.95–14.22× for CellFlow, and 80.28–130.99× for CFM-GP, whose original sampler is extremely fast.
Despite these differences in relative overhead, PROVE adds an average of 0.168–0.659 seconds of sampling time per cell. Therefore, its inference overhead is strongly backbone-dependent and should be assessed using both per-cell latency and relative slowdown.
We thank the reviewer for their careful assessment and constructive feedback. We address each question below using targeted experiments and diagnostic analyses.
Q1. Benefits of Full PROVE over Endpoint-only Correction and Interpolation
Appendix C.4 (Tables 6–8) shows that direct reference transfer is competitive on Replogle. We further compared Full PROVE with endpoint-only correction and backbone–reference interpolation using CellFlow on all three datasets. All methods used identical references and paired sampling seeds (0, 42, 2026). Interpolation weights λ∈0.2,0.4,0.6,0.8,1.0 were selected on 20 training-side pseudo-held-out pairs per dataset using a prespecified perturbation-response score. The selected weights were 0.6/0.6/0.2 for Replogle/Frangieh21/Srivatsan20; no test outcomes informed selection.
Replogle
Frangieh21
Srivatsan20
Across perturbation-specific metrics, Full PROVE exceeds Endpoint-only in 15/15 and validation-tuned Interpolation in 11/15 comparisons, supporting a benefit from stepwise editing beyond one-shot correction or simple reference interpolation.