- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the reliability of the Wasserstein distance as an evaluation metric for single-cell perturbation prediction benchmarks. The core issue is that the metric tends to favor predictions with smaller variance (i.e., "tighter" predictions): even when the predicted cell-expression distribution is not very close to the true distribution, a suitably tight prediction can score better than a sample from the true distribution. The authors provide a theoretical account of this phenomenon and show that tightness substantially affects the metric. They propose PopFloor, a diagnostic procedure that constructs a split-half reference from measured cells, estimates a finite-sample "floor" for Wasserstein distance, and uses it to characterize and quantify the effect of tightness. The authors audit several methods on scPerturBench and PerturBench. They find that removing Wasserstein changes the leaderboard substantially, and that the reproduced entrants are consistently too tight. The paper also offers concrete suggestions, such as additionally reporting energy distance. Overall, these conclusions are relevant to benchmark design and to method selection in this area.
Critical strengths
-
The paper identifies an underappreciated issue in bioinformatics evaluation: excessive trust in the Wasserstein distance. This metric is widely used in single-cell perturbation studies and other single-cell settings, yet its finite-sample behavior is rarely examined systematically. The authors' results show that this issue can substantially affect existing benchmark rankings.
-
The theoretical derivation is clear. The authors prove that, in finite samples, Wasserstein distance systematically favors tighter predictions, and they provide a quantity that can be computed directly from measured cells to quantify the strength of this preference. They further relate this quantity to the prediction dimension: higher-dimensional predictions are more strongly favored when they are tight.
-
The experiments are detailed and the results are convincing. The authors show that existing benchmark rankings are indeed affected by the Wasserstein advantage of tight predictions. Moreover, the top-ranked method's tightness is close to the metric's most preferred tightness, suggesting that its advantage may not reflect genuinely better predictive quality.
Critical weaknesses
-
The paper's scope and impact are limited. There are two specific issues. First, the paper focuses on one evaluation metric in benchmark work for a single prediction task (cell perturbation), and it is unclear how far the conclusions generalize to other tasks or metrics. Second, many methods in this area predict only the mean gene-expression response and rely on the benchmark's completion step to form a cell population. Whether distributional differences matter for such methods is therefore an open question.
-
The proposed "solution" mainly consists of reporting recommendations, such as making the completion rule explicit and reporting energy distance alongside Wasserstein. Within the PopFloor framework, the authors do not provide a more definitive conclusion, especially about when Wasserstein should or should not be trusted. For researchers who hope to use a benchmark to select the best method, the conclusions of this paper can currently serve only as a reference.
Questions for authors
-
The source of tightness remains unclear. For example, the paper finds that CPA's spread is close to the estimator's preferred spread. Is this a coincidence, or is it induced by the training objective or the output convention? I would encourage the authors to briefly discuss whether properties of the prediction method itself are related to this tightness preference.
-
For methods that predict only the mean, are distributional metrics meaningful? If the benchmark separated mean-level and distribution-level leaderboards, or scored such methods only on their predicted means, would this lead to more reasonable results?
-
Most experiments in the paper are designed to show that energy distance is less affected by tightness, but energy distance has its own limitations. Do the authors recommend replacing Wasserstein with energy, or using it only as a complement? When the two rankings disagree, how should a benchmark aggregate or report them?
-
Beyond reporting, could the tightness bias be corrected rather than only diagnosed? Table I.1 already compares differently-routed methods under matched spread, and Section 4.3 rescales CPA to the truth's spread as a single-point intervention. Could these be systematized into a general correction for cell-emitting methods—e.g., scoring all methods at a common spread before computing Wasserstein? What should that common spread be (the truth's, the control's)?
4 comments
Thank you again for your time and thoughtful review. As the discussion deadline approaches, we would be grateful if you could consider our response when your schedule permits. We would be happy to address any remaining questions.
Thank you for the detailed response. The core position is that the paper focuses on diagnosis rather than on correcting method predictions or leaderboard outcomes. I find this framing acceptable and consistent with the paper's original scope. The additional experiments in Appendix M provide concrete evidence that simple variance rescaling is not a viable correction, which strengthens the paper's diagnostic positioning. Overall, the response resolves most of my concerns. I am raising my overall rating to 3 (Weak Accept).
Thank you very much for taking the time to revisit our response and Appendix M, and for your thoughtful feedback throughout the discussion.
Author Response to Reviewer XWWd
Thank you for these questions. PopFloor diagnoses estimator vulnerability rather than certifying a leaderboard (Section 5). Proposition 1 concerns exact equal-size empirical W22 in any task, while our experiments cover single-cell perturbation benchmarks (Appendix H). The preference persists under entropic OT and Sinkhorn divergence on matched arrays (Table A.2). Centered transport D favors the completed linear baseline over CPA in Table 1 but is not proper. Energy V and U penalize the tested collapse controls (Appendices K, L). These checks do not extend Proposition 1 to those estimators. W1 remains untested.
When to trust Wasserstein. For the exact estimator, q>0 means a truth sample loses to its own repeated mean (Corollary 1). This preference is observed on K562 at 100 genes (median ρ=1.50) and Norman19 on all genes (ρ=1.79), but not on Replogle-8 (ρ=0.51) or Norman19 in 30 PCs (ρ=0.66, Tables B.4–B.5). A nonpositive q rules out a strict full-collapse advantage, not all shrinkage. For a>0, s⋆=c/a lies below the variance-matching scale b/a when c<ab. It lies below the native scale 1 only if c<a. Neither condition certifies model rankings (Table M.1), and a below-floor score is not a calibrated false-alarm test.
Source of tightness. CPA's 0.29 spread ratio is close to s⋆=0.27 at 100 genes (Section 4.4), but this does not establish why they are close. The preferred scale moves from 0.71 to 0.06 across 10 to 5,000 genes (Table B.3). scGPT and scouter are also underdispersed (Section 4.3), as are PerturBench's eight models by factors of 3.1 to ∞ (Table 2). In our CPA reproduction, sampling from the fitted Gaussian likelihood instead of emitting its location, with weights fixed, multiplies per-gene spread by about five and puts CPA above the floor on 99.7% of perturbations before and after recentering (Section 4.3, Appendices A and C). Conditional means omit variability represented by the likelihood. Training, representation and decoder effects are not separated, and original CPA dispersion is unmeasured.
Mean-only methods. Separating mean-level and population-level boards clarifies what is evaluated (Section 5, P3). Every method can be scored on predicted means. Because a mean alone defines no cell distribution, scoring it after completion evaluates the model plus the completion rule, and equalizing routing reverses the CPA–linear Wasserstein comparison in all three seeds (Appendix I, Table I.1). Mean-only output does not make biological heterogeneity unimportant.
Recomputing scPerturBench's published per-metric ranks with its pooled aggregation supports separate boards (Appendix M, Table M.3). CPA leads the mean-level board of Pearson and MSE (mean rank 2.97). On the population board of energy, Wasserstein and symmetric KL, GenePert (5.44) and CPA (5.48) are nearly tied, with GenePert first in 64% of 2,000 perturbation resamples. Dropping Wasserstein leaves energy and symmetric KL, where GenePert leads (4.28) and CPA is eleventh of fifteen (8.39), distinct from fourth across all five remaining metrics (Table F.2). Population ranks still score each mean-only model together with its completion rule.
Energy and disagreements. We recommend energy as a complement, not a replacement (Section 5, P2). Energy V and U penalize collapse in all 18 measured-cell settings (Appendix L, Table L.2), yet they probe different geometry than transport, V has O(1/n) bias, and 2–3 permutation and 3 wrong-target controls remain inconclusive. When metrics disagree, benchmarks should report separate effects, matched references and interventions rather than post-hoc weights unless an endpoint fixes aggregation beforehand.
Common-spread scoring. On saved K562 predictions (Appendix M, Tables M.2, M.4), rescaling residuals to calibration-truth or control spread preserves means and residual directions. This is a post-hoc sensitivity probe, not a validated correction. Truth spread uses outcome information, control spread assumes invariant variance, and rescaling removes real variance differences while magnifying tiny residuals (scGPT and scouter have median native variance below 0.3% of target). Both targets worsen W22/b for all three methods at 10, 100 and 1,000 genes without recovering the mean-error ordering, scGPT, scouter, CPA. Under truth spread at 100 and 1,000 genes, CPA leads scGPT by 0.052 [0.040, 0.064] and 0.131 [0.127, 0.135], despite worse mean error. These paired gaps average seeds within each target, then weight targets equally, unlike Table M.2's row-weighted means. Thus equal total variance does not resolve the disagreement, and residual alignment also contributes.