acceptodds
Under review as a conference paper at ICLR 2027

CellShift: Benchmarking Single-Cell Temporal Models on Unseen Biology

Abstract

Single-cell temporal modeling is the task of predicting the gene expression of individual cells at unobserved time points, either after the last measurement (forecasting) or between measurements (interpolation). Machine-learning models such as PRESCIENT, scNODE and CellTempo learn these tasks from single-cell RNA sequencing time courses, which measure thousands of genes in each of thousands of cells. Existing evaluations, however, hold out time points within the cell types and conditions seen during training, so it remains unclear whether these models learn dynamics that transfer to unseen biology. This paper introduces CellShift, a benchmark spanning seven time-course datasets, with three extended to test generalization across held-out cell fates, lineages, and starting populations. It evaluates statistical baselines, neural dynamics models, pretrained temporal models, and language-model agents under a shared evaluator with explicit tracking of information access. On lineage-traced blood development (the LARRY in vitro dataset), PRESCIENT and a population-time MLP outperform simple copy baselines when forecasting the whole day-6 population. However, when evaluated on held-out fates, no fitted model without access to target-day cells improves upon copying the latest observed snapshot by more than 5%, even when initialized from the held-out cells. Even on identical target cells, holding the fate out raises the error of nearly every fitted model. At the clone level, models trained on day-6 states assign at most 28% of predicted cells to a held-out fate that comprises 93 to 95% of the observed population, highlighting the difficulty of generalizing to unseen cell fates. Agents that choose among a fixed set of numerical tools perform on par with a selector that uses no language model.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
single-cell, temporal prediction, trajectory forecasting, benchmark, distribution shift, generalization, lineage tracing, virtual cell, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_VwXDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper establishes a dynamic benchmark for single-cell omics, defines a multi-level set of tasks, and evaluates existing temporal models across these tasks. The authors provide a foundational baseline and conduct comparative analyses, which reveal the performance boundaries of many current methods and highlight several of their imitations.

Critical strengths
  1. The authors have established a multi-level set of tasks and defined various evaluation perspectives, enabling the comparisons of current models from different perspective.

  2. The authors consider a fundamental baseline—replicating data from neighbor time points which, represents a baseline devoid of dynamics. This baseline is used for comparison across the various tasks.

Critical weaknesses
  1. While the paper presents numerous results of existing methods, it fails to distinguish the primary reasons of these failures or provide an in-depth analysis of why various deep learning-based approaches can't handle these tasks. Furthermore, it doesn't offer a constructive direction for future improvement, thereby providing limited guidance on how to further improve models in this field.

  2. As a benchmark study, the paper considers an insufficient number of models. The field of single-cell dynamics is relatively mature, yet the authors have omitted comparisons with many established methods that account for various biological processes—for instance, SF2MSF^2M (which incorporates stochasticity), WFR-FM and DeepRUOT (which account for cell growth and death), Metric FM (which considers data structure), and MMFM or 3MSBM (which account for multi-marginal interplation). Moreover, the study relies on only a single dataset, failing to evaluate performance across other relevant temporal datasets, such as EB, EMT, or mouse. The absence of these classic methods and additional datasets restricts the generalizability and scope of the benchmarking conclusions.

Questions for authors

Based on the aforementioned weaknesses, we raise the following questions and invite the authors to provide clarification and responses:

  1. Could the authors identify common, underlying shortcomings across the failures of the different models? Specifically, what capability is currently lacking in existing methods, and how might this insight offer a direction for future improvements in the field? (Weakness 1)

  2. Could the authors include additional models, particularly classic approaches designed for different biological processes such as $SF^2M$$, Metric FM, DeepRUOT, WFR-FM, MMFM, 3MSBM... It can help provide a more comprehensive assessment of the current methods' operational boundaries? (Weakness 2)

  3. Could the authors provide experimental results on a broader range of datasets? Specifically, by comparing the performance of different models across various datasets (e.g. EMT, EB, mouse...), would it be possible to more thoroughly evaluate the models' generalization capabilities? (Weakness 2)

8 comments

Response to Reviewer VwXD

We thank the reviewer for the detailed feedback and for recognizing the value of our benchmark in evaluating models across different tasks and perspectives. We have built on this framework by evaluating the six suggested models, adding five new studies, and further investigating why models struggle to predict unseen populations. We address each point below.

W1/Q1. Why models fail, and future guidance.

We investigate why models struggle to predict unseen cell populations, using LARRY as a case study. Our analyses suggest two possible reasons: (1) gene expression latent representations may remove information about populations not seen during training, and (2) some widely adopted latent representation methods may fail to capture biologically meaningful relationships among genes. These results describe patterns we observe in the models; further, a (3) biological explanation is proposed below.

Context: Models fail to identify unseen cell fates. In the below table, we demonstrate that even with all genes available, models struggle to identify the correct fate from earlier cells, increasing forecasting difficulty for even later cells. To note, these are not strict forecasting tasks because day-6 cells from other populations remain in training. For the sake of analysis, clones are grouped by their descendants' eventual fate, even if their earlier cells have not yet developed that identity.

Held-out fateActual day-6 cellsCopied earlier cellsMaximum predicted proportion
Neutrophils95%55%47%
Monocytes93%50%25%
Basophils95%77%46%

The proportion of day-6 cells with fate correctly identified. Actual cells use dataset annotations, while predicted cells are labeled via a 15-nearest-neighbor classifier, trained relative to ground truth fate labels. Maximum values are calculated across eleven fitted models, after averaging stochastic configurations over seeds. Actual proportions are below 100% because clones are assigned the fate of most of their day-6 descendants. However, some clones also produce other cell types, and some descendants are still undifferentiated at day 6.

Across all three tasks, models predict a lesser proportion of cells of the actual final fate than the copy baseline. For example, the actual day-6 population comprises 95% neutrophils, compared with 55% in the copied cells and at most 47% in model predictions. This shows that models struggle to capture the developmental changes that distinguish different cell fates, even when population-average expression scores may not fully reflect this error.

Latent representation can distort held-out population information. Several evaluated implementations, including PRESCIENT, OT-CFM, SF²M, Metric FM, DeepRUOT, WFR-FM, and MMFM, compress gene expression using training-fitted PCA. This representation may poorly preserve expression patterns important to a population absent from training. To examine this possibility, we compared a raw-copy control with PCA-reconstructed copies of the same held-out basophil clones’ day-4 cells. The raw-copy control leaves their original gene expression unchanged. The reconstructed controls pass those cells through 50 principal components fitted on either a training set excluding basophils or a size-matched set including separate basophil training clones. No dynamics model is applied: all three outputs are evaluated directly against the same day-6 population. The raw copy achieves RMSE 0.0686, compared with 0.1309 after reconstruction through the basophil-excluded PCA and 0.0728 through the basophil-included PCA—increases of 91% and 6%, respectively. This fixed-seed diagnostic shows that the representation pipeline can worsen the starting cells’ agreement with the future population, particularly when the fate is absent from PCA fitting. Future work should examine how different representations preserve gene-expression information from held-out populations, since this can limit downstream predictions, especially as exhaustively incorporating every possible developmental fate across biological contexts into a latent representation is experimentally impractical.

Restoring lost gene expression does not fix predictions. As an extension of the prior experiment, we then tested whether restoring discarded expressions improves an actual forecast. Keeping MMFM’s trained dynamics fixed, we added back each starting cell’s original expression minus its PCA reconstruction. Basophil RMSE decreased by 21%, from 0.1735 to 0.1367, but remained about twice the error of copying the same starting cells (0.0673). The correction worsened the in-domain and neutrophil forecasts. Thus, restoring discarded gene expression improves predictions, showing that information loss during PCA contributes to prediction errors. This suggests that the latent representations used to organize gene expression may limit generalization to unseen biological dynamics. Future work should explore representations—and model architectures that leverage such representations—that better preserve biological information needed to predict unseen cell biology.

Possible biological explanations. Different cell fates are driven by different gene regulation programs. For example, GATA2 and CEBPA are important for basophil development, while IRF8 and GFI1 play opposing roles in monocyte and neutrophil development. Since these fates share common progenitors, earlier cells may already contain parts of these programs through lineage priming. In principle, models should be able to learn these patterns and predict how cells develop into different fates. However, our results suggest that current models struggle to learn these underlying biological programs and instead rely on patterns from populations seen during training. Gene expression alone may also be insufficient to capture cell fate decisions, which can depend on external signals such as cytokines and cell–cell interactions.

Overall, models should therefore be selected using validation sets that hold out entire biological groups, rather than randomly splitting similar cells between training and validation. This better tests whether models can generalize to unseen biology. Generalization is still possible: scDiffEq outperforms the best copy baseline on two of four held-out Loh genotypes (W3). We will add these analyses to the discussion.

W2/Q2. Coverage of additional models

We added all six requested methods, and five additional methods. These cover different approaches, including modeling cell changes over time (SF²M, scDiffEq), cell growth and death (WFR-FM, DeepRUOT, TIGON, Waddington-OT), cell data structure (Metric FM), predicting multiple time points together (MMFM, 3MSBM), energy-based models (JKOnet*), and a pretrained time-series foundation model (TimesFM).

The table reports RMSE on average gene expression for the same LARRY in vitro targets, averaged over three seeds using the same evaluation method. Each forecast starts from the population's own earlier cells. The first row shows the best copy baseline for each task. For the day-6 forecast, this is the source-population mean used in the manuscript; copying day-4 cells gives 0.0874.

ModelDay-4 interpolationDay-6 forecastHeld-out neutrophilsHeld-out monocytesHeld-out basophils
Best copy baseline0.05370.08500.11190.09180.0657
SF²M0.05280.11130.13470.11430.1478
Metric FM0.05150.21750.23920.25970.1777
DeepRUOT0.04880.13950.11910.12240.1507
WFR-FM0.04880.09730.11760.11570.1482
MMFM0.06090.10840.12940.12640.1735
3MSBM0.1059————
scDiffEq0.05160.06830.10970.10490.1575
JKOnet*0.07300.07730.12780.15320.1688
TIGON0.07030.10790.12690.11200.1502
Waddington-OT0.0594————
TimesFM-3—————

3MSBM and Waddington-OT only support interpolation, so they have no forecasting results or results on the added studies. Waddington-OT uses a version adapted for cell movement over time with uniform growth. TimesFM only predicts average expression and requires at least four earlier time points, so it has no LARRY results and was only tested on Cook and McFarland among the added studies.

On the standard tasks, several added models outperform the best copy baseline by more than 5%: WFR-FM and DeepRUOT reduce error by 9% on day-4 interpolation, while scDiffEq and JKOnet* reduce error by 20% and 9% on day-6 forecasting. However, none of the models improves on the copy baseline by more than 5% on the three held-out fates. scDiffEq comes closest, with 2% lower error on neutrophils.

Across the added models, SF²M, Metric FM, MMFM, WFR-FM, scDiffEq, JKOnet*, and TIGON address all 32 held-out tasks with three seeds. DeepRUOT supports six Cuomo and Hagai tasks. We will include results for each seed, standard deviations, model adaptations, and fixed computational budgets in the appendix.

W2/Q3. Dataset diversity and biological coverage

Our original submission included seven time-course datasets covering embryoid bodies and five mouse datasets. We tested generalization to unseen biological groups across LARRY, CellTag, and Pijuan-Sala. However, LARRY provided most of our controlled analyses, so further testing across more diverse datasets is needed to determine how broadly these findings apply. To address this, we have added five more studies, including EMT, each with one biological group held out from training.

For each task, we average each model's RMSE across seeds and divide it by the best copy-baseline RMSE. The table reports the median ratio across tasks for each study, with the range in parentheses. A ratio below 1 means the model performs better than the copy baseline. To technically clarify, each task contributes to the ratio equally, while the best model can differ between tasks. The ranges show differences across each study’s tasks, not confidence intervals thereof. Also, "Lower" and "higher" mean more than 5% better or worse (not statistically significant differences).

Added studyHeld-out biologyTasksMedian model/copy ratio (range)Versus best copy baseline
Cook EMTCancer cell line95.31× (2.65–7.23×)Higher on all 9
Loh embryoid bodiesGenotype40.93× (0.88–1.00×)Lower on 3; within 5% on 1
Cuomo endodermDonor41.08× (1.03–1.19×)Within 5% on 2; higher on 2
Hagai immune responseStimulus22.28× (1.99–2.56×)Higher on both
McFarland drug responseCancer cell line132.33× (1.61–3.50×)Higher on all 13

The table excludes incomplete runs, pretrained models, and models or preprocessing methods that use target-time cells (i.e. day 6 cells) during training. As in LARRY, we include PRESCIENT, which for validation selects the best checkpoint using target-time cells from non-held-out fates (response to Reviewer Qcnt, Q3). We did not run PRESCIENT on Hagai or Cuomo because of computational cost.

On Loh, PRESCIENT, starting from the held-out genotype's own cells, reduces error by 6.1–11.8% on three genotypes compared with the best copy baseline. scDiffEq also reduces error by 10.3% and 7.7% on two genotypes. Without PRESCIENT, the Loh median ratio is 0.95× (0.90–1.03×), with improvements on two tasks. Cook and McFarland have median ratios of 5.41× and 2.34×, respectively.

To test whether the inability for cross-fate generalization can be attributed to lack of the held-out fate cells in the training set, we compared each model on identical target cells with the held-out fate excluded from training versus returned to training at a matched training size (see 6AEn, W2; Qcnt, W2/Q2). We applied the nine configurations of the LARRY control to every task (six on Hagai and Cuomo, where PRESCIENT has no scored results). We evaluated where the error is more than 5% higher when the held-out fate was excluded in the training set than included, with the condition that the three-seed ranges were non-overlapping. This was consistent in 65 of 81 comparisons on Cook, 9 of 12 on Hagai, 10 of 36 on Loh, and 0 of 24 on Cuomo. McFarland has no matched control, so its higher errors than copying cannot be attributed to withholding the group.

The results suggest models perform worse than the copy baseline on the EMT cell lines, drug-response cell lines, and immune-response conditions. The control experiments also link holding out a biological group to higher errors on Cook and Hagai. However, results on Loh and Cuomo are mixed, and holding out a group has a smaller effect on these studies.

References

[1] Atanackovic, L., et al. (2025). Meta flow matching: Integrating vector fields on the Wasserstein manifold. International Conference on Learning Representations (ICLR).

Reviewer_QcntOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CellShift, a benchmark for evaluating whether single-cell temporal models generalize beyond the biological groups observed during training. It covers seven time-course datasets, with three supporting additional tests on held-out cell fates, lineages, or starting populations. Its main contributions are these biological-shift evaluation protocols, a shared evaluator that explicitly distinguishes information-access regimes, and comparisons spanning simple baselines, neural dynamics models, pretrained temporal models, and language-model agents.

Validation includes temporal interpolation and forecasting, matched comparisons on identical target cells, group-conditioned model variants, and complementary expression, distributional, and cell-fate metrics. The strongest evidence comes from LARRY: models that improve whole-population forecasts generally fail to outperform copying the latest observed cells when predicting held-out fates, even when initialized from those cells. The experiments also demonstrate that low pseudobulk error can conceal unrealistic population composition and that the evaluated agents provide no clear improvement over a non-LLM tool selector. Three-seed experiments and bootstrap analyses support selected findings, while most results outside LARRY use single runs.

Critical strengths
  1. A scientifically relevant benchmark for generalization to unseen biology, supported by targeted controls. CellShift extends conventional time-point holdouts to held-out cell fates, lineages, and starting populations, addressing a central requirement for useful virtual-cell models: transferring learned dynamics beyond the biological groups represented during training. The most persuasive evidence comes from the matched-target comparisons and group-conditioned experiments on LARRY (Tables 15–16). These evaluate identical target cells with and without the fate represented in training and test whether initializing models from the held-out group’s own cells resolves their failures. It generally does not: on held-out basophils, copying the day-4 cells achieves RMSE 0.0657, compared with approximately 0.122–0.160 for the day-4-conditioned dynamics models. Although the broader generality remains uncertain, these controls make the central negative result informative and support the benchmark’s potential value to the community.

  2. A consequential demonstration that favorable aggregate scores can conceal biologically implausible predictions. The paper evaluates pseudobulk expression error alongside distributional distances, cell-fate composition, and marker trajectories. In the target-day-supervised clone track, a repeated training-mean vector achieves an RMSE of 0.0243 but is classified entirely as monocytes, even though the observed population contains substantial neutrophil and basophil fractions. The marker analysis further shows that models can suppress fate-specific expression already present in their starting cells, moving predictions toward the training populations. These findings strengthen the paper beyond a model-ranking exercise: they identify concrete failure modes and explain why apparently accurate forecasts may be unsuitable for biological interpretation. This provides useful guidance for both future benchmark design and model development.

Critical weaknesses
  1. The headline comparisons do not consistently provide models with equivalent context. Several reference models predict a pooled population without conditioning on the held-out group’s observed cells, whereas the test-context copy directly uses those cells. Table 16 substantially addresses this concern through group-conditioned variants, but these variants are excluded from the aggregate win counts in Figure 3b. This makes the headline summary partly reflect differences in conditioning rather than an inability to generalize. Equally conditioned comparisons should form the primary evidence.

  2. The matched-fate controls do not fully isolate biological shift. In Table 15, returning same-fate clones to training changes both biological coverage and training-set size. Although evaluating identical target cells is a substantial strength, the resulting performance gap cannot be attributed exclusively to the presence or absence of the held-out fate. A training-size-matched control is needed because this attribution underpins the paper’s central interpretation of generalization failure.

  3. The breadth of the conclusions exceeds the independent biological-shift evidence. Despite covering seven datasets, the strongest controlled evidence for neural dynamics models comes from three held-out fates within LARRY. The Pijuan-Sala lineage experiment evaluates regression baselines rather than the principal dynamics models, and several other shift tasks lack applicable dynamics-model results. Consequently, the broad claim that current temporal models do not learn transferable dynamics is insufficiently established across biological systems. The conclusions should be narrowed or supported by additional independent, comparably controlled evaluations.

  4. Statistical uncertainty limits confidence in the benchmark-wide rankings. Most experiments outside LARRY use a single run, while some LARRY configurations show substantial seed variability, including selection of an essentially untrained checkpoint. The uniform 5% improvement threshold measures effect size but does not establish statistical reliability. The paired bootstrap analyses are useful, yet resampling saved predictions does not capture uncertainty from retraining or changing training splits. Since model rankings and failure rates are central benchmark outputs, the decisive comparisons need stronger evidence of stability.

Questions for authors
  1. Figure 3b excludes the group-conditioned variants in Table 16. How do aggregate win counts change when these variants are included where applicable, while comparing models with the same context and information-access regime?

  2. In Table 15, the matched control adds same-fate clones to training. Do existing experiments control for the resulting increase in training-set size? If so, does the degradation under fate holdout persist? If not, please clarify how much of the observed gap can reasonably be attributed specifically to biological coverage.

  3. Appendix D states that PRESCIENT selects checkpoints using a “validation window over the interval of the test.” Which time points and populations are available during this selection? In particular, does any result labeled pre-target use target-time observations for checkpoint or hyperparameter selection?

  4. How consistently do the principal held-out-fate findings hold across individual training seeds? Please clarify how failed or essentially untrained checkpoints enter the reported averages and whether the bootstrap intervals incorporate training variability or only resampling of test observations.

7 comments

Response to Reviewer Qcnt

We thank the reviewer for the detailed feedback and suggestions. These helped us strengthen our comparisons with matched starting populations, add a control for training-set size, and expand our evaluation to five additional studies. We also clarified PRESCIENT's use of validation data and added further analyses of model variability and checkpoint selection. We address each point below.

W1/Q1. Fairness of comparisons and Figure 3b

The aggregate win counts do not change when the group-conditioned variants are included. The best fitted model still beats the best copy baseline by more than 5% RMSE on 1 of the 11 held-out-biology tasks, as in the submitted figure, and it is now within 5% on three tasks, compared to two (the neutrophil loss becomes a tie). We agree that equally conditioned comparisons should be the primary evidence. As such, in the revised Figure 3b, the five LARRY tasks with group-conditioned runs will be shown first, and the remaining six tasks will be labeled separately. On each task, Figure 3b compares the lowest fitted-model RMSE with the lowest copy-baseline RMSE, after averaging over seeds.

Table 1 gives the equally conditioned comparison. The copy baseline predicts day 6 with the held-out population's day-4 cells, whereas default PRESCIENT and scNODE start from day-2 cells pooled across populations. Here PRESCIENT and scNODE start from the same day-4 cells as the copy.

Table 1. RMSE of population-average gene expression at day 6, using matched day-4 inference inputs.

Held-out populationCopy of the same day-4 cellsPRESCIENTscNODE
Neutrophils0.11190.1089 ± 0.00170.1251 ± 0.0042
Monocytes0.09180.1149 ± 0.0005†0.1479 ± 0.0056
Basophils0.06570.1216 ± 0.0003†0.1508 ± 0.0061
LK0.09410.1119 ± 0.0036†0.1366 ± 0.0222
LSK0.10200.0652 ± 0.0204†0.0784 ± 0.0078

Mean ± sample standard deviation over three training seeds; lower is better. LK and LSK are two experimentally defined starting populations. †These means include early-checkpoint runs (W4/Q4).

With the same day-4 input, PRESCIENT is within 5% of copying on neutrophils, and neither model is more than 5% below copying on any of the three held-out fates. Both models beat copying on LSK, so transfer is possible for some shifts, although not consistently. Table 1 leaves out OT-CFM because its PCA uses other populations' day-6 cells even when its flow is trained without day 6. Leaving it out of the Figure 3b counts would not change them.

W2/Q2. Training-set size versus biological coverage

No existing experiment controlled for training-set size, so we added this control and found the degradation persists at equal size. The control removes whole clones of the other fates at random until the training cells at days 2 and 4 equal those of the fate-held-out setting. The held-out fate's cells stay in training, and both settings use identical training-cell counts, validation cells, and test cells. Results are in Table 1 of our response to Reviewer 6AEn (W2).

Against the size-matched control, excluding the fate raises RMSE by 7.4% to 36.3% for neutrophils, 16.3% to 51.1% for monocytes, and 7.7% to 60.1% for basophils, across the nine model configurations of Table 15. The configurations are linear time, PCA-ridge, population-time MLP, and PRESCIENT and scNODE each started from pooled day-2 cells, the held-out population's day-2 cells, or its day-4 cells. The gap is clear (above 5%, with non-overlapping seed ranges for seeded models) in 26 of the 27 comparisons. The exception is PRESCIENT started from day-4 cells on neutrophils (+7.4%, overlapping seed ranges).

To estimate how much of the original gap reflects biological coverage, we re-evaluated the Table 15 control together with the new version. In the 23 comparisons where it shows a clear gap, at least 93% of that gap remains at matched size. Removing clones of the other fates also raises the held-out fate's share of the control's training cells (for neutrophils from 22% to 28%), which can make the matched gap larger than the original one. We will provide the complete per-seed results in the appendix.

W3. Biological coverage and the scope of the conclusions

Motivated by the reviewer's suggestion to expand independent held-out-biology evaluations, we incorporated five additional studies into the benchmark, covering 32 held-out tasks.

StudyPrediction settingHeld-out axisTasks
Cook EMTHuman cancer-cell responses to signals inducing cell-state changesCell line9
CuomoHuman stem-cell differentiationDonor4
HagaiInnate immune responsesStimulus2
LohMouse embryoid-body differentiation under genetic perturbationsGenotype4
McFarlandHuman cancer-cell responses to trametinibCell line13

These evaluations include three-seed scNODE experiments across all 32 tasks, alongside dynamics models newly added during the rebuttal. Complete model coverage is reported in our response to Reviewer VwXD (W2).

We compare the lowest seed-mean RMSE among completed fitted models with the best copy baseline on each task. The counts exclude incomplete runs, pretrained methods, and models or preprocessing fitted on target-time cells. As on LARRY, PRESCIENT is included although it selects checkpoints on target-time cells of other groups (Q3). The best model is more than 5% below the best copy baseline on three tasks, within 5% on three, and more than 5% higher on 26. All three improvements are on Loh. There, PRESCIENT started from the held-out genotype's own cells outperforms the best copy baseline on three genotypes, and the newly added scDiffEq beats it on two. The effect of excluding the group itself is tested with matched controls in our response to Reviewer VwXD (W3).

The dynamics models now also run on the Pijuan-Sala lineage task. All of them lose to copying the held-out lineage's own cells, and the best, scDiffEq, has 20% higher error.

Because models transfer on Loh, we will state the conclusions only for the evaluated models and studies. In the discussion, we will replace

"Current temporal models do not transfer to unseen fates."

with

"There is evidence that temporal generalization is limited, but performance depends on the biological shift context."

Moreover, the discussion will distinguish the controlled LARRY findings from the broader evaluations and explain that the fixed-budget, PCA-based experiments do not establish a universal limit on what temporal models can learn.

Finally, we will also replace the abstract's statement

"However, when evaluated on held-out fates, no fitted model without access to target-day cells improves on simply copying the latest observed snapshot by more than 5%, even when initialized from the held-out cells, and on identical target cells, holding the fate out raises the error of nearly every fitted model."

with

"In LARRY, no fitted model outperforms the copy baseline for held-out fates by more than 5%, even when initialized from the latest observed cells, although models do transfer to a held-out starting population. On identical test cells and at matched training-cell counts, excluding the fate from training still increases error. Across five additional studies, improvements over copying remain limited and context dependent."

Q3. PRESCIENT validation windows and target-time access

Yes. Some results labeled "pre-target" use target-time validation observations for checkpoint selection. PRESCIENT fits dynamics using training cells from days 2 and 4, but selects checkpoints using day-6 measurements from separate validation clones. In the held-out-fate experiments, those validation cells belong to non-held-out fates, and the held-out fate's future test measurements remain hidden.

Further, the regression baselines select hyperparameters by cross-validation using only the training time points. PRESCIENT runs in Table 1 use fixed hyperparameters. We will identify the observations used for fitting, selection, and inference separately in Appendix D and the access tables. Where validation targets are unavailable, checkpoints are selected using training loss.

W4/Q4. Variability across training seeds and checkpoint selection

The principal LARRY finding is consistent across three seeds. On the three LARRY held-out fates, no seed of day-4-conditioned PRESCIENT or scNODE—or of the eight added forecasting models—beats the copy baseline by more than 5% when initialized from the held-out population's day-4 cells. We will report individual errors alongside the means and label single-run results separately.

Seeded models now have three training seeds on LARRY and on the five added studies. On the added studies, population-time MLP, scNODE, and OT-CFM each have three seeds across all 32 held-out tasks and 19 controls, and PRESCIENT has three seeds on Cook, Loh, and McFarland, where it was run.

Some PRESCIENT runs trained for 200 epochs, but validation restored weights were saved after only 1 to 3 epochs. We call these early-checkpoint runs. In Table 1 they are marked † and are all three seeds of day-4-conditioned PRESCIENT on monocytes and basophils and one seed each on LK and LSK. The small variation across seeds on monocytes and basophils can therefore reflect repeated selection of an early model. On the 32 held-out tasks of the added studies, they are 81 of the 234 PRESCIENT runs.

We count every run, as in the submission, and mark early-checkpoint runs so that readers can set them aside. Leaving them out changes no LARRY count in W1 and no count in W3. Additionally, failed runs are reported as failures and never enter the averages.

Bootstrap intervals resample test observations from saved predictions and do not include retraining variability. We will report these sources of uncertainty separately. We keep the 5% threshold and describe it as an effect-size convention without a significance claim. To note, since computational constraints limit these experiments to three training seeds on fixed splits, uncertainty across alternative splits and biological replicates remains unmeasured.

Reviewer_6AEnOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new benchmark for evaluating how single cell temporal models generalize to unseen biological settings. The authors evaluate several models across seven datasets including settings where cell fates or lineages are held out during training. They find that models often fail to outperform simple copying the last snapshot baselines under these shifts.

Critical strengths
  • The paper addresses an interesting gap in prior temporal benchmarks.
  • Broad experimental evaluation. Experiments cover several datasets, methods and evaluation settings.
Critical weaknesses

My background is primarily in machine learning rather than biology, so some of the following concerns may come from my limited understanding of the biological setting. I would appreciate clarification on the following points.

  • Motivation for cross fate generalization: My main concern is why we should expect dynamics learned from some cell fates to generalize to entirely different ones. Is there a biological evidence suggesting that this should be possible? Otherwise, the negative results would be expected and do not necessarily indicate a failure of the models.

  • Confounding effects of removing training data: Holding out a fate also reduces the number of samples network has seen during training. Could the observed performance drop partly be explained by this?

  • Real world motivation: I am not entirely sure what practical scenario the cross-fate setting represents. In LARRY groups seem to be defined using their eventual fate, but in a real forecasting problem that information may not even be available. It would help to explain how this evaluation relates to a practical prediction task.

  • Fairness of comparisons. Most models seem to be designed for predicting how an observed biological process evolves time (interpolate or extrapolate over time), rather than to generalize across unseen cell fates. For this reason it could be unfair to judge these models on their capability of something they are not designed for.

  • LLM agents: Could the authors clarify the motivation for including LLM agents in this benchmark? Is the hypothesis that they can perform better numerical tool selection or that their pretrained biological knowledge can help predict unseen dynamics?

Questions for authors

See weaknesses.

5 comments

Response to Reviewer 6AEn

We thank the reviewer for the thoughtful questions about CellShift's biological motivation, evaluation design, and practical relevance. These questions helped us clarify the value of cross-fate evaluation, add a control for training-set size, and expand our experiments on LLM-based forecasting. We address each point below.

W1. Motivation for cross-fate generalization

Biology gives reason to expect at least partial transfer across fates. Cells from the same progenitor (starting) population develop into distinct fates (specialized cell types) through shared mechanisms, including gene regulation and responses to extracellular signals. A comprehensive virtual-cell model aims to represent these mechanisms across developmental outcomes [2], so predicting a fate it has not seen is part of its purpose. Our cross-fate evaluation asks whether dynamics learned from related populations predict the later gene-expression states of a fate excluded from training. W3 below describes the practical prediction tasks this stands for.

LARRY provides a concrete setting for this question. Neutrophils, monocytes, and basophils are distinct blood cell types that develop from related progenitors under the same culture conditions. Weinreb et al. [1] showed that cells developing toward these fates share early developmental structure before their trajectories diverge.

Our results show transfer in some settings. Models that started from the held-out LSK population, a sorted group of early blood progenitors, outperformed the copy baseline (response to Reviewer Qcnt, Table 1). On an added embryoid-body study, a newly added model beats the best copy baseline on 2 of 4 held-out genotypes (response to Reviewer VwXD, W3). However, it’s worth emphasizing that no model beats copying a held-out fate by more than 5%.

W2. Confounding effects of training-set size

The smaller training set affects the results minimally. Because the control in the manuscript (Table 15) includes held-out fates in the training set—and more cells in the training set as a result—we added a control that keeps the training-set size fixed. For each held-out fate, it contains cells of that fate and fewer cells of the other fates, with the same number of training cells at days 2 and 4 as the held-out setting. Both settings use identical validation and test cells.

Table 1. Increase in RMSE when a fate is excluded from training.

Held-out fateAgainst the original control (more training cells)Against the size-matched control (same number)
Neutrophils−1.0% to 23.7%7.4% to 36.3%
Monocytes16.8% to 28.6%16.3% to 51.1%
Basophils6.1% to 60.5%7.7% to 60.1%

Ranges over the nine configurations of Table 15 in the manuscript: linear time, PCA-ridge, population-time MLP, and PRESCIENT and scNODE each started from pooled day-2 cells, the fate's own day-2 cells, or its day-4 cells. We reran the original control together with the new one. Two PRESCIENT variants differ from the printed Table 15 by 2 to 9 percentage points, within their spread over seeds.

Holding the fate out still raises the error at equal training size. To technically clarify, we call a gap clear when it is above 5% and, for seeded models, the seed ranges do not overlap. The gap is clear in 26 of the 27 comparisons. The exception is PRESCIENT started from day-4 cells on neutrophils (+7.4%, overlapping seed ranges). In the 23 comparisons where the original control shows a clear gap, at least 93% of that gap remains at matched size. Removing clones of the other fates also raises the held-out fate's share of the control's training cells (for neutrophils from 22% to 28%), which can make the matched gap larger than the original one. Further details are in our response to Reviewer Qcnt (W2), and the full comparison will be added to the appendix.

W3. Real-world motivation and fate-label availability

The LARRY fate tasks are a controlled test of whether learned dynamics transfer to a biologically different group. The reviewer is right that these groups are defined after the fact. For LARRY, a clone's fate is read from its day-6 cells, and a real forecast would not know in advance which day-4 cells will become, e.g. basophils. However, fate labels are never model inputs; instead, fate labels are used to determine which clones are held out and which cells are evaluated, and in the runs started from the fate's own cells they also determine the starting population of cells—all of which are related to experimental setup.

A practical case involves determining how a starting population of cells of a specific group—such as a cell line, a donor, a genotype, or a stimulus—evolve over time. For example, in drug discovery, a drug response measured over time in several cell lines could be used to forecast the response of such a drug in a new cell line before it is actually experimentally measured; practically, this minimizes more time-consuming biological analysis with computational analysis. The five studies we added follow this design (response to Reviewer VwXD, W3). Such forecasts matter because measuring every cell line or condition is impractical, and virtual-cell models are meant to predict ones that were not measured [2].

W4. Fairness of evaluating models on unseen fates

We agree that most evaluated models were originally developed for temporal interpolation or forecasting. However, cross-fate evaluation reflects an inherent feature of the biological processes these models seek to represent. Cells from related progenitor populations can diverge toward distinct fates through shared regulatory and signaling mechanisms. Therefore, assessing whether learned dynamics transfer across these developmental outcomes is a biologically meaningful test of temporal single-cell modeling and its usefulness for virtual-cell simulation [2].

We apply established models from the literature to their existing prediction tasks of interpolation and forecasting. In this way, the prediction objective—estimating gene-expression states at unobserved times—remains the same. We extend the evaluation by excluding particular biological populations from training, allowing us to assess whether the models’ temporal predictions generalize beyond the biology represented in their training data.

W5. Motivation for LLM agents

The submitted experiments tested the first hypothesis, tool selection. Agents chose among a fixed library of numerical tools and performed about as well as a deterministic selector using the same library. The reviewer's question motivated new experiments on the second hypothesis, in which agents write their own forecasting algorithms and may draw on pretrained biological knowledge.

In these experiments, agents inspected the permitted training data and observed context, chose their own methods, and wrote and executed code within ten-minute sessions. Preprocessing and parameter selection were restricted to training data, while external resources and hidden targets were prohibited. We evaluated GPT-6.1 Sol at high reasoning effort and DeepSeek V4.1 Flash at maximum reasoning effort.

Table 2. LARRY day-6 forecasting without day-6 training cells, pseudobulk RMSE (error of population-average expression) ↓. New experiments are bolded.

MethodApproachRMSE ↓
PRESCIENTNeural dynamics0.0519 ± 0.0116
Population-time MLPRegression0.0600 ± 0.0016
GPT-6.1 Sol (high effort)Algorithm generation0.0613 ± 0.0172
Source-population meanCopy baseline0.0850
Pooled persistence (copying day 4)Copy baseline0.0874
Deterministic tool selectorFixed-tool selection0.0946
Single agent (DeepSeek V4.1 Flash)Fixed-tool selection0.0953 ± 0.0015
DeepSeek V4.1 Flash (max effort)Algorithm generation0.0965 ± 0.0074

No method trains on day-6 cells. PRESCIENT selects its checkpoint on day-6 cells of separate validation clones (response to Reviewer Qcnt, Q3), and computational budgets differ. Three algorithm-generating sessions were attempted per model. Sol's result includes all three, while DeepSeek's includes two after code review found one run that fitted PCA on the test population's input cells, violating training-only preprocessing. This exclusion was decided before test scoring. Repeated runs report mean ± sample standard deviation.

GPT-6.1 Sol outperformed both copy baselines on average and approached the population-time MLP. We extended this algorithm-generation experiment to the three LARRY held-out-fate forecasting tasks already reported in the manuscript, retaining their task definitions, permitted observations, and evaluator.

Table 3. Algorithm-generating agents on existing held-out-fate tasks, pseudobulk RMSE ↓. Parentheses give the number of included runs.

Held-out fateSol (high effort)DeepSeek (max effort)Copy of the fate's day-4 cells
Neutrophil0.1064 ± 0.0143 (3 runs)0.1268 ± 0.0196 (3 runs)0.1119
Monocyte0.0935 ± 0.0255 (3 runs)0.1359 ± 0.0019 (2 runs)0.0918
Basophil0.1139 ± 0.0551 (2 runs)0.1468 (1 run)0.0657

Before test scoring, one Sol basophil run was excluded for choosing a decision threshold with the test population's input cells. DeepSeek's missing results are sessions that did not deliver predictions within the deadline (one monocyte and two basophil sessions). These tasks permit other fates' day-6 training cells.

GPT-6.1 Sol had lower mean error than DeepSeek on every fate, but neither agent improved on the copy baseline by more than 5%. To test the role of biological knowledge, we repeated Sol's day-6 forecast with gene names replaced by anonymous identifiers.

Table 4. Gene-name ablation, Sol (high effort), pseudobulk RMSE ↓.

Gene identifiersRMSE ↓
Original gene symbols0.0613 ± 0.0172
Anonymized identifiers0.0675 ± 0.0115

Each condition has three runs, with numerical inputs, gene order, prompt structure, environment, validation policy, and resource limits held fixed.

Hiding gene names raised Sol's mean error by about 10%, within the spread across runs, and both conditions outperformed the copy baselines on average. A benefit from recognizing genes is therefore not shown, and the results are consistent with general numerical reasoning. With three runs per condition and evaluation limited to LARRY, these findings remain exploratory.

References

[1] Weinreb, C., et al. (2020). Lineage tracing on transcriptional landscapes links state to fate during differentiation. Science, 367, eaaw3381.

[2] Bunne, C., et al. (2024). How to build the virtual cell with artificial intelligence: Priorities and opportunities. Cell, 187, 7045–7063.

Reviewer_gJUXOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

na

Critical strengths

na

Critical weaknesses

na

Questions for authors

na

1 comment

Response to Reviewer gJUX

We thank the reviewer for their time and consideration of our work. Our general response lists the new experiments and the changes to the paper. We would be happy to provide any additional clarifications or address any questions that may arise during the discussion period.

Comments on the paper

1 comment

General response to all reviewers

We thank the reviewers for their careful reading. Reviewers noted that CellShift addresses a gap in temporal benchmarks (6AEn) and evaluates broadly across datasets, methods and information regimes (6AEn, VwXD). They also noted that targeted controls on LARRY support its central result and that low pseudobulk error can hide implausible cell-type composition (Qcnt). The main requests were to separate training-set size from biological coverage, compare models with equal context, add models and datasets, explain why models fail, and narrow the claims. Below we list what we added and what changes in the paper.

New experiments

  • Size-matched control (6AEn W2, Qcnt W2/Q2). At equal training size, holding a fate out still raises the error in 26 of 27 comparisons, and at least 93% of the original gap remains.
  • Equal-context comparison and Figure 3b (Qcnt W1/Q1). With the group-conditioned variants included, the win counts do not change (1 of 11 held-out-biology tasks). Started from the same day-4 cells as the copy, neither PRESCIENT nor scNODE beats copying a held-out fate by more than 5%. Both transfer to the held-out LSK starting population.
  • Eleven added models (VwXD W2/Q2). All six requested methods (SF²M, Metric FM, DeepRUOT, WFR-FM, MMFM, 3MSBM) and five more (scDiffEq, JKOnet*, TIGON, Waddington-OT, TimesFM). Several beat copying on the standard LARRY tasks, and none beats copying a held-out fate by more than 5%.
  • Five added studies (VwXD W2/Q3, Qcnt W3). EMT, embryoid bodies, endoderm differentiation, immune stimuli and drug response, with 32 held-out tasks and matched controls on four studies. The best model beats the best copy baseline on 3 tasks, all on embryoid bodies, is within 5% on 3, and loses on 26.
  • Why models fail (VwXD W1/Q1). Predicted day-6 populations contain fewer cells of the held-out fate than the copied starting cells. The PCA representation loses information about a fate it was not fitted on, as a matched control shows. Adding the lost expression back improves MMFM's basophil forecast by 21% but does not reach the copy baseline. We give directions for future representations, models and validation splits.
  • Seeds and checkpoints (Qcnt W4/Q4). The principal LARRY finding holds in every seed. Seeded models have three training seeds on LARRY and the added studies, and runs that kept an early checkpoint are marked. Leaving them out changes no LARRY or added-study count.
  • LLM agents (6AEn W5). We added agents that write their own forecasting code, and a gene-name ablation. None of the experiments outperform the copy baseline by more than 5%, and ablating gene identities does not substantially change performance—indicating generated algorithms may not be greatly biologically informed.

Changes to the paper

  • Figure 3b and its caption include the group-conditioned variants, with the five equal-context LARRY tasks shown first.
  • The discussion heading "Current temporal models do not transfer to unseen fates" and the corresponding abstract sentence are replaced with narrower statements (Qcnt W3).
  • The size-matched control, the rerun of Table 15, the five added studies, the added models and all per-seed results are added to the appendix. After the rerun, the range "7 to 88%" in the results and discussion becomes "6 to 85%".
  • Table 23 lists all fourteen clone-level methods, and the abstract's "at most 28%" becomes "at most 47%".
  • The discussion gains the failure analysis and the directions for future models (VwXD W1/Q1).
  • Appendix D and the access tables state the observations used for fitting, checkpoint selection and inference (Qcnt Q3).

We will upload a revised PDF with these changes marked.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.