- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
this paper proposed AmbiModBench for single-cell gene perturbation prediction benchmark to address 1) conventional absolute metrics cannot identify shared versus target-specific perturbation response, 2) some metrics are less sensitive to gene-coordinate permutation thus cannot be used to accurately measure gene-level prediciton, and 3) train split doesn't sufficiently describe the represetation space of pertubration in held-out split. thus they proposed 1) comparing model prediction results to training-mean baseline on same train split, 2) using permutation test and independent replicate calibration to check sensitivity of metrics against gene identity and measurement noise, and 3) by defining ambiguity modulus and test-to-training embedding radius to derive conditional converge bound based on Lipschitz continuity.
Critical strengths
- compared model predictions and training-mean baseline on same target held-out to avoid judging whether a model learnt target-dependent information by high absolute score. their rigorous design on target-disjoint train/test split and clusterwise target bootstrap provide more statistical evidence on the results.
- separately discussing gene aligned readouts and marginal response distributions explains why measurement noise can might turn, e.g. MSE towards collapsing to cohort mean. by calibrating on gene permutation and independent-replicate, it provdies more useful diagnostics for reference.
Critical weaknesses
- the paper didn't propose much non-incremental or novel methodological improvement. there has been plenty of studies proposing similarly systematic studies regarding shared perturbation response, uninformative baseline, or perturbation-specific evaluation. Training-mean baseline, psotiive or negative controls, control-relative performance comparisons and introducing confidence interval were proposed in predated literatures. and no non-trivial new conclusion are revealed by their benchmark comparing to existing results, thus it leverages the significance for its benchmark contribution effort.
- top 100 direction agreement seems only checked predicted signs for ground-truth top response genes, rather than requires accurate identification of genes under perturbation nor punishing FP prediction of other genes.
- reconducting PCA on each train split raise discrepancy on representation space under different fractions, thus the conclusion that coverage determines gain rather than sample size cannot be sufficiently supported.
Questions for authors
- can u provde direct empirical evidence to prove embedding coverage is boosting paired predictive performance? e.g. show distance distribution between target embedding held-out to their nearest target embedding in train split?
- can u compare perfromance on train subsets with different coverage while maintaining train split size the same?
- can u reason about the choice to compare embedding distance under different fractions that will go through separate PCA process?
- can u add independent replicate calibration analysis, and assess affected-gene idnetification and direction correctness?