acceptodds
Withdrawn from ICLR 2027

Diagnosing Representation and Prediction Errors in Virtual-Cell Models under Distribution Shift

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Predicting transcriptional responses to unseen drugs and cellular contexts is a central challenge in virtual-cell research. Overall prediction errors do not distinguish limited response-space coverage from inaccurate prediction within the covered space. We present a diagnostic framework that separates these components using response spaces learned from training data and characterizes within-space errors through prediction direction and magnitude. Analyses of LINCS and MoABox show that substantial response coverage can coexist with limited prediction accuracy. On MoABox, the response space captures 81.9% of target energy, yet most error from a direct predictor occurs within this space. Cross-cell experiments further show that fixed magnitude shrinkage improves transfer on LINCS and OP3 over unscaled predictions while preserving direction. The framework provides a clearer account of where virtual-cell predictions fail and how their errors change under specific model adjustments.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Virtual cells, Perturbation response prediction, Out-of-distribution generalization, Response-space coverage, Error decomposition

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ycRLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes failure modes of perturbation-response prediction under distribution shift. Its main idea is to use a response subspace estimated from the training data to distinguish limited response-space coverage from inaccurate prediction within the covered space. The authors further examine prediction direction and magnitude, and evaluate these diagnostics across multiple drug- and cell-OOD settings.

Critical strengths

I find the diagnostic angle of the paper interesting. Separating limited response-space coverage from prediction error within the covered space provides a useful way to reason about why perturbation-response models fail under distribution shift, rather than relying only on aggregate prediction metrics.

Critical weaknesses
  1. The main limitation is that the proposed framework is primarily diagnostic and descriptive. The core analysis relies on a relatively straightforward decomposition of prediction error with respect to a training-derived low-rank response space, and does not introduce a substantially new statistical testing methodology or a deeper theoretical framework for characterizing distribution shift. More importantly, the analysis stops short of providing actionable guidance for improving perturbation-response models. Although the paper identifies whether errors arise from limited response-space coverage or inaccurate prediction within the covered space, it does not show how these diagnostics should translate into new training objectives, model architectures, or learning strategies, nor does it demonstrate that using the proposed diagnosis leads to improved models.

  2. The empirical analysis is mainly based on a few simple, internally constructed predictor variants and memory-based baselines, rather than established modern perturbation-response models. Since the proposed diagnostics can in principle be applied to arbitrary full-response predictors by projection into the learned response space, it is unclear why representative modern virtual-cell models are not included. Without such evaluations, the claimed generality of the framework remains insufficiently supported.

Questions for authors
  1. Could the authors clarify whether the proposed diagnostic framework can be applied to established single-cell perturbation-response models, rather than only to the simple predictor variants used in this paper?

  2. How can the proposed diagnostics concretely guide the training or design of current single-cell perturbation-response models?

  3. The proposed framework appears to be mainly post-hoc, since response-space coverage requires access to the ground-truth perturbation response. How do the authors envision using this framework in prospective settings, where the response to a new perturbation has not yet been measured? Could coverage or the likely source of prediction failure be estimated from model inputs alone?

Reviewer_sWSsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a diagnostic framework that separates representation coverage from prediction accuracy in virtual-cell models. Experiments across five datasets reveal substantial within-space prediction errors and show that magnitude calibration can improve cross-cell transfer.

Critical strengths
  1. The framework provides an interpretable decomposition of representation and prediction errors, offering insights beyond aggregate performance metrics.
  2. Experiments cover multiple datasets and OOD settings, with detailed protocols and uncertainty estimates.
  3. The analysis reveals that better directional alignment does not necessarily improve prediction accuracy, highlighting the importance of magnitude calibration.
Critical weaknesses
  1. The core error decomposition follows directly from standard orthogonal projection. Its methodological novelty and added value over existing subspace evaluation methods remain unclear.
  2. The evaluation primarily uses relatively simple predictors, it remains unclear whether the diagnostic findings generalize to other perturbation-response models, such as compositional, generative, and optimal-transport-based approaches.
  3. Although the paper demonstrates several model adjustments, it does not clearly establish how the proposed diagnostics provide actionable guidance beyond conventional evaluation metrics.
Questions for authors
  1. How can the proposed diagnostics provide actionable guidance for model development beyond identifying where prediction errors occur?
  2. What factors might explain the inconsistent effects of direction supervision and magnitude calibration across different datasets?
  3. How broadly can the framework be applied to different virtual-cell modeling approaches, such as generative and optimal-transport-based models?
Reviewer_QpkKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce a framework for diagnosing generalizability of models to out-of-distribution samples. The framework uses two types of models: Chemical OOD and Cell OOD. Using these two models, the authors evaluate the learned latent space for 5 datasets: LINCS, SciPlex3, Cellarity, OP3, and Novartis MoABox.

Critical strengths

The authors evaluate generalizability to out-of-distribution drugs and cells on 5 datasets.

The two types of models for Chemical OOD and Cell OOD seem to be simple, almost representative workflows of existing models for drug/perturbation response prediction.

Critical weaknesses

The framework aims to diagnose challenges with OOD generalization, but there is no evidence that state-of-the-art models indeed have challenges with generalizability to OOD samples.

The two types of models are overly simplified. It is okay as a proof-of-concept / controlled demonstration of the framework. But there is no validation of state-of-the-art models to demonstrate to what extent the models are able to generalize to OOD drugs and cells.

Questions for authors

Please include experiments with state-of-the-art models to analyze the extent of their limited OOD generalizability.

Please clarify the contributions of this work given the overly simplified models in the framework, lack of analysis on state-of-the-art models, and lack of comparison against existing frameworks/models that evaluate OOD generalizability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.