- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper analyzes failure modes of perturbation-response prediction under distribution shift. Its main idea is to use a response subspace estimated from the training data to distinguish limited response-space coverage from inaccurate prediction within the covered space. The authors further examine prediction direction and magnitude, and evaluate these diagnostics across multiple drug- and cell-OOD settings.
Critical strengths
I find the diagnostic angle of the paper interesting. Separating limited response-space coverage from prediction error within the covered space provides a useful way to reason about why perturbation-response models fail under distribution shift, rather than relying only on aggregate prediction metrics.
Critical weaknesses
-
The main limitation is that the proposed framework is primarily diagnostic and descriptive. The core analysis relies on a relatively straightforward decomposition of prediction error with respect to a training-derived low-rank response space, and does not introduce a substantially new statistical testing methodology or a deeper theoretical framework for characterizing distribution shift. More importantly, the analysis stops short of providing actionable guidance for improving perturbation-response models. Although the paper identifies whether errors arise from limited response-space coverage or inaccurate prediction within the covered space, it does not show how these diagnostics should translate into new training objectives, model architectures, or learning strategies, nor does it demonstrate that using the proposed diagnosis leads to improved models.
-
The empirical analysis is mainly based on a few simple, internally constructed predictor variants and memory-based baselines, rather than established modern perturbation-response models. Since the proposed diagnostics can in principle be applied to arbitrary full-response predictors by projection into the learned response space, it is unclear why representative modern virtual-cell models are not included. Without such evaluations, the claimed generality of the framework remains insufficiently supported.
Questions for authors
-
Could the authors clarify whether the proposed diagnostic framework can be applied to established single-cell perturbation-response models, rather than only to the simple predictor variants used in this paper?
-
How can the proposed diagnostics concretely guide the training or design of current single-cell perturbation-response models?
-
The proposed framework appears to be mainly post-hoc, since response-space coverage requires access to the ground-truth perturbation response. How do the authors envision using this framework in prospective settings, where the response to a new perturbation has not yet been measured? Could coverage or the likely source of prediction failure be estimated from model inputs alone?