acceptodds
Under review as a conference paper at ICLR 2027

PopFloor: Split-Half Diagnostics for Finite-Sample Wasserstein Bias in Single-Cell Perturbation Benchmarks

Abstract

Single-cell perturbation models are evaluated by comparing predicted and measured populations. Yet the plug-in Wasserstein distance has a finite-sample floor and can fall when a prediction shrinks toward its mean, so underdispersed predictions can outscore a target sample. PopFloor estimates this split-half floor from measured cells. By an exact per-draw identity for the unregularised squared 2-Wasserstein distance, a spread-free population beats an equal-size target sample wherever the floor's ratio to the held-out variance exceeds one plus the relative mean error. In an exploratory recomputation, scPerturBench's leader falls to fourth without Wasserstein distance. Our reproduced CPA scores below the floor on 88.7 to 92.6% of the largest screen's perturbations in three seeds, and by the pre-registered rule beats it on three of five screens. The ratio and that share rise together from 10 to 5,000 genes under the exact estimator, as pre-registered, while its relative mean error falls. Pre-registered tests show the ratio's margin over that threshold predicts, beyond gene and cell counts, the preference on disjoint cells of two unused screens and, on the reproductions, entrants' advantage over an independent floor in 11 of 12 runs. PopFloor shows where Wasserstein rankings can reward underdispersion.

Primary area:
datasets and benchmarks
Keywords:
single-cell perturbation, benchmark evaluation, Wasserstein distance, finite-sample bias, underdispersion, distribution comparison, scPerturBench

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XWWdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the reliability of the Wasserstein distance as an evaluation metric for single-cell perturbation prediction benchmarks. The core issue is that the metric tends to favor predictions with smaller variance (i.e., "tighter" predictions): even when the predicted cell-expression distribution is not very close to the true distribution, a suitably tight prediction can score better than a sample from the true distribution. The authors provide a theoretical account of this phenomenon and show that tightness substantially affects the metric. They propose PopFloor, a diagnostic procedure that constructs a split-half reference from measured cells, estimates a finite-sample "floor" for Wasserstein distance, and uses it to characterize and quantify the effect of tightness. The authors audit several methods on scPerturBench and PerturBench. They find that removing Wasserstein changes the leaderboard substantially, and that the reproduced entrants are consistently too tight. The paper also offers concrete suggestions, such as additionally reporting energy distance. Overall, these conclusions are relevant to benchmark design and to method selection in this area.

Critical strengths
  1. The paper identifies an underappreciated issue in bioinformatics evaluation: excessive trust in the Wasserstein distance. This metric is widely used in single-cell perturbation studies and other single-cell settings, yet its finite-sample behavior is rarely examined systematically. The authors' results show that this issue can substantially affect existing benchmark rankings.

  2. The theoretical derivation is clear. The authors prove that, in finite samples, Wasserstein distance systematically favors tighter predictions, and they provide a quantity that can be computed directly from measured cells to quantify the strength of this preference. They further relate this quantity to the prediction dimension: higher-dimensional predictions are more strongly favored when they are tight.

  3. The experiments are detailed and the results are convincing. The authors show that existing benchmark rankings are indeed affected by the Wasserstein advantage of tight predictions. Moreover, the top-ranked method's tightness is close to the metric's most preferred tightness, suggesting that its advantage may not reflect genuinely better predictive quality.

Critical weaknesses
  1. The paper's scope and impact are limited. There are two specific issues. First, the paper focuses on one evaluation metric in benchmark work for a single prediction task (cell perturbation), and it is unclear how far the conclusions generalize to other tasks or metrics. Second, many methods in this area predict only the mean gene-expression response and rely on the benchmark's completion step to form a cell population. Whether distributional differences matter for such methods is therefore an open question.

  2. The proposed "solution" mainly consists of reporting recommendations, such as making the completion rule explicit and reporting energy distance alongside Wasserstein. Within the PopFloor framework, the authors do not provide a more definitive conclusion, especially about when Wasserstein should or should not be trusted. For researchers who hope to use a benchmark to select the best method, the conclusions of this paper can currently serve only as a reference.

Questions for authors
  1. The source of tightness remains unclear. For example, the paper finds that CPA's spread is close to the estimator's preferred spread. Is this a coincidence, or is it induced by the training objective or the output convention? I would encourage the authors to briefly discuss whether properties of the prediction method itself are related to this tightness preference.

  2. For methods that predict only the mean, are distributional metrics meaningful? If the benchmark separated mean-level and distribution-level leaderboards, or scored such methods only on their predicted means, would this lead to more reasonable results?

  3. Most experiments in the paper are designed to show that energy distance is less affected by tightness, but energy distance has its own limitations. Do the authors recommend replacing Wasserstein with energy, or using it only as a complement? When the two rankings disagree, how should a benchmark aggregate or report them?

  4. Beyond reporting, could the tightness bias be corrected rather than only diagnosed? Table I.1 already compares differently-routed methods under matched spread, and Section 4.3 rescales CPA to the truth's spread as a single-point intervention. Could these be systematized into a general correction for cell-emitting methods—e.g., scoring all methods at a common spread before computing Wasserstein? What should that common spread be (the truth's, the control's)?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer XWWd

Thank you for these questions. PopFloor diagnoses estimator vulnerability rather than certifying a leaderboard (Section 5). Proposition 1 concerns exact equal-size empirical W22W_2^2 in any task, while our experiments cover single-cell perturbation benchmarks (Appendix H). The preference persists under entropic OT and Sinkhorn divergence on matched arrays (Table A.2). Centered transport DD favors the completed linear baseline over CPA in Table 1 but is not proper. Energy V and U penalize the tested collapse controls (Appendices K, L). These checks do not extend Proposition 1 to those estimators. W1W_1 remains untested.

When to trust Wasserstein. For the exact estimator, q>0q>0 means a truth sample loses to its own repeated mean (Corollary 1). This preference is observed on K562 at 100 genes (median ρ=1.50\rho=1.50) and Norman19 on all genes (ρ=1.79\rho=1.79), but not on Replogle-8 (ρ=0.51\rho=0.51) or Norman19 in 30 PCs (ρ=0.66\rho=0.66, Tables B.4–B.5). A nonpositive qq rules out a strict full-collapse advantage, not all shrinkage. For a>0a>0, s⋆=c/as^\star=c/a lies below the variance-matching scale b/a\sqrt{b/a} when c<abc<\sqrt{ab}. It lies below the native scale 1 only if c<ac<a. Neither condition certifies model rankings (Table M.1), and a below-floor score is not a calibrated false-alarm test.

Source of tightness. CPA's 0.29 spread ratio is close to s⋆=0.27s^\star=0.27 at 100 genes (Section 4.4), but this does not establish why they are close. The preferred scale moves from 0.71 to 0.06 across 10 to 5,000 genes (Table B.3). scGPT and scouter are also underdispersed (Section 4.3), as are PerturBench's eight models by factors of 3.1 to ∞\infty (Table 2). In our CPA reproduction, sampling from the fitted Gaussian likelihood instead of emitting its location, with weights fixed, multiplies per-gene spread by about five and puts CPA above the floor on 99.7% of perturbations before and after recentering (Section 4.3, Appendices A and C). Conditional means omit variability represented by the likelihood. Training, representation and decoder effects are not separated, and original CPA dispersion is unmeasured.

Mean-only methods. Separating mean-level and population-level boards clarifies what is evaluated (Section 5, P3). Every method can be scored on predicted means. Because a mean alone defines no cell distribution, scoring it after completion evaluates the model plus the completion rule, and equalizing routing reverses the CPA–linear Wasserstein comparison in all three seeds (Appendix I, Table I.1). Mean-only output does not make biological heterogeneity unimportant.

Recomputing scPerturBench's published per-metric ranks with its pooled aggregation supports separate boards (Appendix M, Table M.3). CPA leads the mean-level board of Pearson and MSE (mean rank 2.97). On the population board of energy, Wasserstein and symmetric KL, GenePert (5.44) and CPA (5.48) are nearly tied, with GenePert first in 64% of 2,000 perturbation resamples. Dropping Wasserstein leaves energy and symmetric KL, where GenePert leads (4.28) and CPA is eleventh of fifteen (8.39), distinct from fourth across all five remaining metrics (Table F.2). Population ranks still score each mean-only model together with its completion rule.

Energy and disagreements. We recommend energy as a complement, not a replacement (Section 5, P2). Energy V and U penalize collapse in all 18 measured-cell settings (Appendix L, Table L.2), yet they probe different geometry than transport, V has O(1/n)O(1/n) bias, and 2–3 permutation and 3 wrong-target controls remain inconclusive. When metrics disagree, benchmarks should report separate effects, matched references and interventions rather than post-hoc weights unless an endpoint fixes aggregation beforehand.

Common-spread scoring. On saved K562 predictions (Appendix M, Tables M.2, M.4), rescaling residuals to calibration-truth or control spread preserves means and residual directions. This is a post-hoc sensitivity probe, not a validated correction. Truth spread uses outcome information, control spread assumes invariant variance, and rescaling removes real variance differences while magnifying tiny residuals (scGPT and scouter have median native variance below 0.3% of target). Both targets worsen W22/bW_2^2/b for all three methods at 10, 100 and 1,000 genes without recovering the mean-error ordering, scGPT, scouter, CPA. Under truth spread at 100 and 1,000 genes, CPA leads scGPT by 0.052 [0.040, 0.064] and 0.131 [0.127, 0.135], despite worse mean error. These paired gaps average seeds within each target, then weight targets equally, unlike Table M.2's row-weighted means. Thus equal total variance does not resolve the disagreement, and residual alignment also contributes.

Thank you again for your time and thoughtful review. As the discussion deadline approaches, we would be grateful if you could consider our response when your schedule permits. We would be happy to address any remaining questions.

Thank you for the detailed response. The core position is that the paper focuses on diagnosis rather than on correcting method predictions or leaderboard outcomes. I find this framing acceptable and consistent with the paper's original scope. The additional experiments in Appendix M provide concrete evidence that simple variance rescaling is not a viable correction, which strengthens the paper's diagnostic positioning. Overall, the response resolves most of my concerns. I am raising my overall rating to 3 (Weak Accept).

Thank you very much for taking the time to revisit our response and Appendix M, and for your thoughtful feedback throughout the discussion.

Reviewer_64zjOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

PopFloor diagnoses finite-sample Wasserstein preferences for underdispersed predictions. An exact identity for equally sized empirical populations expresses squared Wasserstein distance as a quadratic in prediction spread, yielding a measured-cell threshold for when a repeated mean beats a truth sample. The paper connects this diagnostic to benchmark rankings through reproduced models, controlled spread/completion interventions, and prediction on disjoint calibration and evaluation cells.

Critical strengths
  1. A simple quantitative diagnostic connects theory to practice. The identity and threshold turn a known qualitative failure mode into a measurable condition. The tests on disjoint cells of previously unused screens support predictive information beyond gene and cell counts, rather than merely restating an identity on the same samples.
  2. The interventions identify a consequential evaluation mechanism. Holding predictions' means or model parameters fixed while changing spread or likelihood sampling shows how output construction can reverse comparisons. This provides useful evidence for improving benchmark interpretation, beyond observing that different metrics produce different rankings.
Critical weaknesses
  1. The causal connection to the published leaderboard remains incomplete. Original cell-level predictions are unavailable, and the CPA reproduction agrees with published mean-level orderings on only 30.3–37.8% of perturbations (§4.2; Appendix F). The missing embedding can affect both means and dispersion. Dropping Wasserstein establishes metric dependence of the published ranking; the interventions establish a shrinkage mechanism in the reproductions. They do not establish how much of the original winner's advantage was caused by that mechanism. This limits the strongest benchmark-level interpretation, although it does not invalidate the diagnostic.
Questions for authors
  1. Assessment-critical robustness: How sensitive are the spread/completion ranking conclusions to a CPA reproduction that more closely matches the published mean-level performance? This would determine how far the mechanism demonstrated here transfers to the original leaderboard.
  2. Assessment-critical analysis: On the saved predictions, does the calibration margin predict spread-induced reversals between competing methods beyond cell counts and mean-error differences? Appendix K.5 predicts margins relative to a floor, which is a different target from comparative model ordering.
1 comment
AuthorsOpenReview ↗

Author Response to Reviewer 64zj

Thank you for drawing this distinction, which the revision states explicitly across Sections 4.2, 4.3 and 6 and Appendices C and F. Recomputing the published table establishes metric dependence of the published leaderboard. Spread, completion and likelihood-sampling interventions establish an underdispersion mechanism in our reproductions. Neither identifies what fraction of the original winner's advantage came from underdispersion, and both of your assessment-critical questions remain open at that boundary.

CPA fidelity. On K562, our CPA reproduction matches the published CPA's ordering against the linear baseline on 98.5–100% of perturbations under Wasserstein distance (and 99.7–100% under symmetric KL and 84.8–89.5% under energy), but on only 30.3–37.8% under mean-level metrics, where the published CPA beats the linear baseline on 96.9–99.4% of perturbations versus 28.5–37.5% for ours (Section 4.2, Appendix F). The missing external gene embedding can affect both means and residuals through CPA's nonlinear decoder (Appendix C). Substituting an oracle mean would not recover the original model, and no closer-fidelity CPA result is available. Under Proposition 1, W^(s)=Δ+as2+b−2cs\widehat{W}(s) = \Delta + a s^2 + b - 2cs, so varying spread with residuals fixed changes a single model's score independently of Δ\Delta. Lowering Δ\Delta alone lowers its exact score and cannot reduce its below-floor share (Appendix F). Pairwise ranking after rescaling or equalizing completion depends on both the mean-error gap and the original residuals (a,c)(a,c), which remain unmeasured. scGPT and scouter, reproduced from public checkpoints and embeddings with 99.1–100% Wasserstein ordering agreement, and PerturBench's own CPA are also underdispersed (Sections 4.3 and 4.6, Appendices C, D and G). These findings do not test transfer to the published scPerturBench CPA.

Pairwise reversals beyond Appendix K.5. Replacing each reproduced method's K562 cell population by copies of its own mean frequently reverses pairwise orderings under exact equal-size W22W_2^2, while whether calibration qq predicts which comparison reverses is unresolved (Section 4.5, Appendix M, Table M.1). Keeping means and trained weights fixed, we score saved CPA, scGPT and scouter predictions natively and after full collapse across 569 distinct perturbations from three overlapping splits (248, 225 and 231 eligible targets per seed) at 10, 100 and 1,000 genes. Each pair has 2,112 non-independent records. Three technical partitions are averaged, each target's records share one of five folds, and 1,000 target-bootstrap refits form the pointwise intervals. The base linear-probability predictor uses log10 gene count, log10 half-cell count and its square, and the calibration mean-error difference, while the extended predictor adds qq from disjoint measured calibration cells. Positive Brier gain (base minus plus-qq) favors adding qq.

PairReversals / observationsBrier gain [95% interval]
CPA–scGPT1,043 / 2,112+0.000525 [−0.001115, +0.002271]
CPA–scouter723 / 2,112+0.000033 [−0.001054, +0.001092]
scGPT–scouter200 / 2,112−0.000089 [−0.000396, +0.000110]

All three 95% intervals cross zero. At a fixed 0.5 threshold, adding qq changes 213 predictions to correct and 208 to incorrect for CPA–scGPT, 1 and 5 for CPA–scouter, and 0 and 0 for scGPT–scouter. On the continuous collapse-induced gap change, adding qq lowers target-weighted MAE by 0.00174 and 0.00197 for the first two pairs and raises it by 0.000044 for the third, which does not establish reliable reversal classification.

Scope. This post-hoc K562 test examines full collapse under exact equal-size W22W_2^2. Rescaling the same predictions to a common spread also fails to recover the mean-level ordering (Appendix M, Tables M.2 and M.4). Transfer to the entropic estimator, other screens and biological model quality remains untested, so PopFloor diagnoses estimator vulnerability without providing a general selector among trained models.

Reviewer_ym4QOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when finite-sample Wasserstein evaluation favors underdispersed predictions of single-cell perturbation responses. It introduces PopFloor, a diagnostic based on a measured-cell split reference and a normalized margin, and derives an exact spread-dependent identity for the unregularized empirical squared W₂ distance. The authors test the diagnostic on disjoint cells and audit benchmark rankings using controlled changes to prediction spread and completion.

Critical strengths
  • S1. A meaningful and interpretable diagnostic. The paper identifies a practically important evaluation failure: in some finite-sample settings, an overly concentrated population can score better than a sample from the target distribution. The cell-only margin q gives a concrete condition for when a repeated-mean copy beats a truth sample under the stated estimator. On two previously unused screens, q measured on calibration cells predicts the corresponding preference on disjoint cells beyond gene and cell counts.

  • S2. Extensive experiments examine the phenomenon from several angles. The authors combine analyses across multiple screens with independent-cell validation, controlled spread and completion interventions, alternative metrics, simulations, and a second benchmark. Holding the mean fixed in the spread intervention is particularly helpful for isolating the effect on scores.

Critical weaknesses
  • W1. The practical value of q beyond existing multi-metric checks remains unclear. The manuscript already reports matched split references and energy distance, along with mean-level metrics such as MSE and Pearson distance and comparisons of predicted and observed gene-wise spread. These complementary checks can flag some concentrated predictions, although each measures a different property and does not automatically diagnose finite-sample Wasserstein bias. For the exact unregularized, equal-size estimator when reference variance is nonzero, q > 0 is equivalent to a repeated-mean control scoring below the matched real-sample reference; with that control, the same extreme-collapse preference is directly observable. The P and Q experiments show that adding q improves prediction beyond gene and cell counts, but they do not directly quantify how often q changes an audit decision when the matched reference, energy distance, and other available checks are already considered.

Overall assessment: I lean toward acceptance (Rating: 3 / 4).

Questions for authors
  • Quantitative comparison. Across held-out perturbations, could the authors compare pre-specified audit rules based on a matched split reference and energy distance, optionally including mean-level MSE or Pearson distance and predicted-to-observed gene-wise spread, against the same checks with q added?
  • Concrete held-out case. Could the authors provide an example showing whether and how q changes an audit conclusion relative to the matched split reference and energy comparison, perhaps alongside mean-level error or the spread ratio?
1 comment
AuthorsOpenReview ↗

Author Response to Reviewer ym4Q

Thank you for raising this practical question. Once a matched split-half truth reference and its mean-gap component are available on calibration cells, qq is a closed-form summary rather than an independent signal. Its incremental Brier gain beside the matched reference and energy distance is unresolved (Section 6, Appendix M). Its demonstrated predictive use is anticipatory, before any entrant is trained or scored. Computed from measured calibration cells in the pre-registered P study, adding qq beyond gene and cell counts lowers mean absolute error in predicting the normalized truth-sample collapse margin on disjoint cells by 0.082 [0.068, 0.095] on Adamson (72 targets) and 0.453 [0.439, 0.466] on Replogle-7 (104 targets) using frozen regressions fitted on other screens, and by 0.052 [0.039, 0.064] and 0.059 [0.047, 0.071] when refitted across target folds within each screen (Section 4.4, Appendix K.4, Table K.4). For equal-size unregularized empirical W22W_2^2 with reference variance b>0b>0 and repeated-mean score Wrep=b+ΔW_{\mathrm{rep}} = b + \Delta, Corollary 1 gives

q=F−Wrepb=Fb−1−Δb,q = \frac{F - W_{\mathrm{rep}}}{b} = \frac{F}{b} - 1 - \frac{\Delta}{b},

so q>0q > 0 is equivalent to Wrep<FW_{\mathrm{rep}} < F, and qq adds no information once F/bF/b and Δ/b\Delta/b are recorded on those cells. Neither result establishes biological model selection.

Comparison with a native check panel. A post-hoc analysis of saved K562 reproductions (Appendix M) tests whether calibration qq helps predict when replacing a model's predicted population by copies of its own mean lowers exact W22W_2^2 on disjoint evaluation cells. This endpoint measures metric behavior rather than biological accuracy. The dataset spans 569 distinct perturbations across three overlapping splits (248, 225 and 231 eligible targets per seed) at 10, 100 and 1,000 genes, giving 2,112 non-independent target, seed and gene-count records per method. Three technical partitions are averaged, each target's records share one of five folds, and 1,000 target-bootstrap refits give pointwise 95% intervals. The native panel uses calibration-cell W22/bW_2^2/b, squared mean error$/b$, energy-V$/\sqrt{b}$, delta-Pearson and its undefined-value indicator, residual-variance ratio, log10 gene count and a quadratic in log10 half-cell count in target-weighted linear-probability fits clipped to [0, 1].

MethodOut-of-fold Brier reduction [pointwise 95% interval]Alert changes at 0.5
CPA0.000184 [0.000023, 0.000479]0
scouter0.001242 [0.000465, 0.002250]17 (11 to correct, 6 to incorrect)

Each method has 2,112 records. Because scGPT has only one positive collapse event, it supports no discrimination claim. An expanded panel that also includes the matched truth split's F/bF/b and Δ/b\Delta/b reconstructs qq algebraically, so we treat its zero increment as an identity rather than an empirical estimate with a confidence interval.

The matched-reference and energy panel you describe. A second post-hoc check, with its interpretation fixed before running, adds the matched truth split's W22/bW_2^2/b and energy-V$/\sqrt{b}$ to the native panel while omitting the truth split's mean-gap component Δ/b\Delta/b, so qq is not reconstructed. Adding qq gives an out-of-fold Brier reduction (base minus plus-qq, positive favoring qq) of −0.0000006 [−0.0000076, +0.0000036] for CPA and −0.000019 [−0.000134, +0.000100] for scouter (Appendix M). Both intervals cross zero. The unresolved gain establishes neither incremental decision value beside those checks nor redundancy.

Concrete held-out case. ACTR1A (seed 1, 100 genes, 28 cells per half) is a post-hoc descriptive example. On calibration cells, F/b=1.697165F/b = 1.697165 and Δ/b=0.084942\Delta/b = 0.084942, giving q=0.612223q = 0.612223 and flagging collapse vulnerability for a dispersed truth sample. Existing checks already flag CPA's concentration, with residual-variance ratio 0.044544 and calibration energy$/\sqrt{b}$ of 0.395411 versus the truth split's 0.109768. Because CPA is already tightly concentrated, collapsing it further on disjoint evaluation cells raises W22/bW_2^2/b from 0.992742 to 1+0.042409=1.0424091 + 0.042409 = 1.042409 (+0.049667) and worsens normalized energy by 0.234221. A positive truth-calibration qq diagnoses the estimator's preference against truth-level spread rather than predicting gain from collapsing an already concentrated model, and this case produces no qq-driven alert change.

Scope. These new checks cover K562, exact equal-size W22W_2^2 and a metric-behavior endpoint. Decision gains over a panel containing the matched reference, transfer to the benchmark's entropic estimator and biological model selection remain unshown.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.