- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines perturbation discrimination scores (PDS) using identity-free predictions, matched permutation nulls, and inversion of a known shrinkage map. It asks whether a corrected PDS reflects model predictions or information introduced during scoring.
Critical strengths
1.The identity-free control and matched nulls provide a clear way to test whether a high PDS contains perturbation-specific directional information.
2.The methodology and reporting are careful. The paper distinguishes support-derived from truth-derived inputs and provides a label-free inverse control for a known empirical-Bayes map.
Critical weaknesses
1.The practical implication of the shrinkage analysis is unclear. A high raw PDS still indicates that the final output distinguishes the correct perturbation. Inverse recovery shows that an internal estimate is recoverable, but not why the lower PDS of the deployed shrunk output should change its interpretation.
2.The empirical scope is narrow: five Perturb-seq panels and an external archive reusing three screens. Moreover, the inverse control requires an explicit empirical-Bayes map, so its relevance to typical black-box predictors is unclear.
Questions for authors
1.On a common candidate library and gene space, how often do raw PDS, normalized PDS, and matched-null conclusions change external model rankings?
2.When a normalization uses the true response of the evaluated perturbation, should the resulting score be removed, labeled as a conditional metric, or reported alongside raw PDS?
3.For black-box predictors without an explicit invertible map, which components of the framework remain applicable, and do they change conclusions beyond raw PDS and response-error metrics?
7 comments
Thank you for the detailed rebuttal and additional analyses, which have helped clarify several of my earlier concerns.
The common-library experiments demonstrate that different scoring rules can affect model rankings, and these changes primarily reflect scoring sensitivity. I also appreciate the clarifications regarding truth-dependent normalization and the framework's applicability to black-box predictors.
Nevertheless, I remain concerned about the incremental value and practical significance of the proposed diagnostic framework. While the analyses identify potential limitations in interpreting raw PDS, it remains unclear how shrinkage works well in practice. Overall, although the rebuttal has addressed some of my concerns, I have decided to maintain my original score and confidence level.
Shrinkage accuracy and the evaluation decision
Thank you for pinpointing what remained unclear. In our experiments, shrinkage works well in practice by substantially reducing prediction error on noisy Perturb-seq profiles, and that accuracy gain is precisely what creates the retrieval scoring dilemma our framework addresses:
-
How shrinkage works well in practice (Appendix E, p. 21; Table E.1, p. 22):
- Consistent error reduction: Empirical-Bayes shrinkage achieves lower squared error and MAE than the unshrunk -shot mean in all 30 dataset-by-support-size settings, cutting squared error by 5.7× to 12.5× at one support cell. Even at (Table E.1), it beats the unshrunk mean on Pearson correlation, MAE, and DE sign accuracy across all 5 panels (e.g., raising DE sign accuracy from 0.540 to 0.856 on Norman and from 0.593 to 0.792 on Arc-H1).
- Opposite raw PDS outcome: Despite losing on accuracy, the noisy unshrunk mean ranks 1st in all 10/10 raw PDS comparisons (five panels across L1 and L2), while every shrinking, projecting, or trained estimator wins 0/10. As you noted, raw PDS still accurately describes retrieval by the deployed output; it simply penalizes contraction toward the shared cellular response that improves prediction accuracy.
-
Practical decisions changed by our black-box controls (Sections 4–5, pp. 5–8, no inverse or retraining needed): Because error-reducing predictors lose raw PDS, recent benchmarks turn to post-hoc normalizations or mean-baseline comparisons. Our controls prevent two concrete benchmark misinterpretations:
- Catching test-norm leakage in post-hoc "fixes": Matching each prediction to its evaluated target's true magnitude lifts a zero-information constant prediction by up to +0.165 above chance (Table 1), allowing effectively constant
scBERTarchive outputs to beat the mean on both K562 splits (Section 5, p. 7). - Separating magnitude alignment from gene-specific direction on raw PDS: Even on unmodified raw L1 PDS,
GEARSbeats the mean by +0.0652 on 384-target RPE1 split 2, yet fails to exceed either its global or magnitude-stratified direction-null (Appendix M) while scoring 0.043 below the same-library mean on top-20 effect overlap (Appendix Z.3).
- Catching test-norm leakage in post-hoc "fixes": Matching each prediction to its evaluated target's true magnitude lifts a zero-information constant prediction by up to +0.165 above chance (Table 1), allowing effectively constant
- Practical takeaway: Keep raw PDS alongside prediction error and effect recovery, label own-truth normalization as a truth-conditioned diagnostic, and require direction-null clearance before crediting a mean-beating gain to learned perturbation identity.
Author Response to Reviewer uZxc
Thank you for these questions. On common candidate libraries, normalization changes point rankings, but conditional intervals resolve few reversals. Scoring-input controls apply to black-box predictors without an inverse. None of this invalidates the raw PDS of a deployed shrunk output. Inverting a known map tests precursor recoverability and does not imply that its score should replace the deployed output's score.
1. Common-library rankings and null conclusions. In new Appendix Z.2 (Table Z.2), we reanalysed the original GEARS, scGPT, self-trained linear and mean predictions across six contexts sharing one candidate library and 1,000 genes per context (Adamson 21/21, K562 107/108, RPE1 171/164), reproducing all 16,576 retained per-perturbation values exactly across the three reused screens.
Each transition below spans 36 model-pair/context comparisons.
Intervals are exploratory and conditional on fixed predictions and libraries, so point flips need not be resolved reversals. Direct paired L1 gap changes exclude zero in 11/36 raw-to-training-fixed and 18/36 raw-to-own-truth comparisons, measuring gap shifts rather than resolved flips. For scGPT minus mean on common-library RPE1 split 1 (N=171), the gap moves from +0.04808 [0.02551, 0.07257] under raw L1 to −0.01594 [−0.03201, −0.00034] under own-truth L1, a paired change of −0.06402 [−0.08577, −0.04210], showing scoring sensitivity rather than which ordering is biologically right.
Matched-null status on common libraries is separate (Tables L.3, L.5, L.7). Raw-to-training-fixed L1 changes 1/18 non-mean statuses, GEARS on 164-target RPE1 split 2 from inside (0.5625 in [0.5610, 0.5701]) to below (0.4955 vs [0.4990, 0.5071]). Raw-to-own-truth L1 changes 1/18, scGPT on Adamson split 1 (N=21) from inside (0.5692 in [0.4807, 0.5714]) to above (0.6349 vs [0.4853, 0.6009]). We do not rank by null subtraction, and the 78-pair method-specific libraries cannot replace common-library rankings.
2. Reporting truth-dependent normalization. We recommend keeping it only alongside raw PDS, response error and effect recovery, labeled a truth-conditioned diagnostic with its identity-free control and matched direction-null. Because it reads the receiving perturbation's held-out norm before forming distances, its discrimination cannot be credited solely to deployment-available predictions.
3. Black-box use and the resulting decision. Input declaration, identity-free controls, matched global and stratified permutations, and effect-recovery reporting need only predictions and aligned candidate responses, not a known map. Inversion is omitted when the map is unknown, since our inverse applies only to the explicit empirical-Bayes map and the noisy training-pair proxy fit passed in 23/30 settings, failing its all-settings criterion (Appendix P.1).
These controls change conclusions beyond raw PDS and response error alone. On the full 384-target RPE1 split 2, GEARS beats the mean on raw L1 by 0.0652 [0.0371, 0.0944] (0.5665) yet lies below its global direction-null [0.5666, 0.5723] and inside its stratified null (Table M.3), and beats the mean on own-truth L1 by 0.0223 [0.0145, 0.0302] (0.7856) inside [0.7818, 0.7872] while losing 0.043 on top-20 effect overlap (Appendix Z.3), whereas scBERT's effectively constant predictions beat the mean on K562 only via own-truth norms (Appendix M). We retain the reported gains while withholding directional attribution, without invalidating raw PDS or claiming biological signal is absent.
Numerical correction and stability. Repeating the 78-pair family under 20 new 2,000-permutation streams leaves 464/468 rule-by-reference cells and every stratified own-truth class unchanged (Appendix Z.1). Separately, on Adamson split 2 (N=13), the RPE1-embedding linear model has an exact zero own-truth bootstrap lower bound on the PDS rank lattice rather than 8.54e-18, lowering own-truth mean-beaters from 32 to 31 (27 above global null, 25 above stratified, while raw and training-fixed stay 31 and 32, Table M.1). Evidence remains scoped to the five internal panels and three reused external screens (including post-hoc top-20 effect recovery in Appendix Z.3, where within-pair score-overlap medians are 0.59 and 0.50 for training-fixed and own-truth versus 0.22 for raw L1).