- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies single-cell perturbation prediction, which identifies probability misallocation between the learned distribution and the true data distribution: even when the model can predict the mean expression precisely, it can still assign wrong probability mass to different expression levels. The paper separates errors at the zero boundary from errors in the positive-expression region using survival function and KS-based diagnostics. It then evaluates these errors across PBMC, Replogle, and Tahoe100M, using modeling paradigms including Linear, CPA, Squidiff, STATE, and Doloris*. Informed by the diagnostics, the paper proposes 3 correction strategies, including hurdle parameterization, incidence supervision, and multi-threshold supervision. Experiments show that direct probability supervision can reduce probability misallocation, as well as improve metrics on differential expressions and perturbation effects. The paper also analyzes why several learning objectives can still leave probability misallocation, and how the choice of supervision is related to sparsity and error location.
Critical strengths
-
The paper conducts a systematic empirical analysis to characterize probability misallocation. The experiments are well controlled by using a shared preprocessing, split, and evaluation protocol across several modeling paradigms on 3 datasets. It also compares prediction KS against a real-real reference. This makes the identified failure mode convincing.
-
The correction experiments reasonably isolated the effect of probability supervision, by keeping the backbone and main training setup fixed.
Critical weaknesses
- The analysis does not yet support a general explanation of the failure mechanism across objectives.
Section 5 gives useful mechanism-specific explanations for mean supervision, generative objectives, and distribution matching, but these results have different scopes, with some of them being local diagnostics or counterexamples. Thus, the broader claims about why existing objectives leave probability misallocation should be stated more carefully.
- The supervision-choice result does not yet show how to choose a better supervision before testing.
The analyses on sparsity and error location explain the observed difference between incidence and multithreshold supervision, but they are descriptive analyses rather than selection rules. Although this is explicitly acknowledged, a rule that can choose the better supervision before seeing test performance would be appealing.
Questions for authors
-
Could you clarify which results only show that an objective can leave probability misallocation, and which results support that the objective itself systematically tends to produce this failure?
-
Could you provide a prospective test of the supervision-choice rule? Based on your findings, do you have any insights on this?
-
Do you expect incidence or multi-threshold supervision to work similarly for other paradigms such as CellFlow or PerturbDiff, as the correction experiments mainly focus on STATE, Doloris*, and the Gaussian predictor? If this is not tested, should the broader claim be restricted to the model families evaluated with these corrections?