- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper establishes a dynamic benchmark for single-cell omics, defines a multi-level set of tasks, and evaluates existing temporal models across these tasks. The authors provide a foundational baseline and conduct comparative analyses, which reveal the performance boundaries of many current methods and highlight several of their imitations.
Critical strengths
-
The authors have established a multi-level set of tasks and defined various evaluation perspectives, enabling the comparisons of current models from different perspective.
-
The authors consider a fundamental baseline—replicating data from neighbor time points which, represents a baseline devoid of dynamics. This baseline is used for comparison across the various tasks.
Critical weaknesses
-
While the paper presents numerous results of existing methods, it fails to distinguish the primary reasons of these failures or provide an in-depth analysis of why various deep learning-based approaches can't handle these tasks. Furthermore, it doesn't offer a constructive direction for future improvement, thereby providing limited guidance on how to further improve models in this field.
-
As a benchmark study, the paper considers an insufficient number of models. The field of single-cell dynamics is relatively mature, yet the authors have omitted comparisons with many established methods that account for various biological processes—for instance, (which incorporates stochasticity), WFR-FM and DeepRUOT (which account for cell growth and death), Metric FM (which considers data structure), and MMFM or 3MSBM (which account for multi-marginal interplation). Moreover, the study relies on only a single dataset, failing to evaluate performance across other relevant temporal datasets, such as EB, EMT, or mouse. The absence of these classic methods and additional datasets restricts the generalizability and scope of the benchmarking conclusions.
Questions for authors
Based on the aforementioned weaknesses, we raise the following questions and invite the authors to provide clarification and responses:
-
Could the authors identify common, underlying shortcomings across the failures of the different models? Specifically, what capability is currently lacking in existing methods, and how might this insight offer a direction for future improvements in the field? (Weakness 1)
-
Could the authors include additional models, particularly classic approaches designed for different biological processes such as $SF^2M$$, Metric FM, DeepRUOT, WFR-FM, MMFM, 3MSBM... It can help provide a more comprehensive assessment of the current methods' operational boundaries? (Weakness 2)
-
Could the authors provide experimental results on a broader range of datasets? Specifically, by comparing the performance of different models across various datasets (e.g. EMT, EB, mouse...), would it be possible to more thoroughly evaluate the models' generalization capabilities? (Weakness 2)
8 comments
Restoring lost gene expression does not fix predictions. As an extension of the prior experiment, we then tested whether restoring discarded expressions improves an actual forecast. Keeping MMFM’s trained dynamics fixed, we added back each starting cell’s original expression minus its PCA reconstruction. Basophil RMSE decreased by 21%, from 0.1735 to 0.1367, but remained about twice the error of copying the same starting cells (0.0673). The correction worsened the in-domain and neutrophil forecasts. Thus, restoring discarded gene expression improves predictions, showing that information loss during PCA contributes to prediction errors. This suggests that the latent representations used to organize gene expression may limit generalization to unseen biological dynamics. Future work should explore representations—and model architectures that leverage such representations—that better preserve biological information needed to predict unseen cell biology.
Possible biological explanations. Different cell fates are driven by different gene regulation programs. For example, GATA2 and CEBPA are important for basophil development, while IRF8 and GFI1 play opposing roles in monocyte and neutrophil development. Since these fates share common progenitors, earlier cells may already contain parts of these programs through lineage priming. In principle, models should be able to learn these patterns and predict how cells develop into different fates. However, our results suggest that current models struggle to learn these underlying biological programs and instead rely on patterns from populations seen during training. Gene expression alone may also be insufficient to capture cell fate decisions, which can depend on external signals such as cytokines and cell–cell interactions.
Overall, models should therefore be selected using validation sets that hold out entire biological groups, rather than randomly splitting similar cells between training and validation. This better tests whether models can generalize to unseen biology. Generalization is still possible: scDiffEq outperforms the best copy baseline on two of four held-out Loh genotypes (W3). We will add these analyses to the discussion.
W2/Q2. Coverage of additional models
We added all six requested methods, and five additional methods. These cover different approaches, including modeling cell changes over time (SF²M, scDiffEq), cell growth and death (WFR-FM, DeepRUOT, TIGON, Waddington-OT), cell data structure (Metric FM), predicting multiple time points together (MMFM, 3MSBM), energy-based models (JKOnet*), and a pretrained time-series foundation model (TimesFM).
The table reports RMSE on average gene expression for the same LARRY in vitro targets, averaged over three seeds using the same evaluation method. Each forecast starts from the population's own earlier cells. The first row shows the best copy baseline for each task. For the day-6 forecast, this is the source-population mean used in the manuscript; copying day-4 cells gives 0.0874.
| Model | Day-4 interpolation | Day-6 forecast | Held-out neutrophils | Held-out monocytes | Held-out basophils |
|---|---|---|---|---|---|
| Best copy baseline | 0.0537 | 0.0850 | 0.1119 | 0.0918 | 0.0657 |
| SF²M | 0.0528 | 0.1113 | 0.1347 | 0.1143 | 0.1478 |
| Metric FM | 0.0515 | 0.2175 | 0.2392 | 0.2597 | 0.1777 |
| DeepRUOT | 0.0488 | 0.1395 | 0.1191 | 0.1224 | 0.1507 |
| WFR-FM | 0.0488 | 0.0973 | 0.1176 | 0.1157 | 0.1482 |
| MMFM | 0.0609 | 0.1084 | 0.1294 | 0.1264 | 0.1735 |
| 3MSBM | 0.1059 | — | — | — | — |
| scDiffEq | 0.0516 | 0.0683 | 0.1097 | 0.1049 | 0.1575 |
| JKOnet* | 0.0730 | 0.0773 | 0.1278 | 0.1532 | 0.1688 |
| TIGON | 0.0703 | 0.1079 | 0.1269 | 0.1120 | 0.1502 |
| Waddington-OT | 0.0594 | — | — | — | — |
| TimesFM-3 | — | — | — | — | — |
3MSBM and Waddington-OT only support interpolation, so they have no forecasting results or results on the added studies. Waddington-OT uses a version adapted for cell movement over time with uniform growth. TimesFM only predicts average expression and requires at least four earlier time points, so it has no LARRY results and was only tested on Cook and McFarland among the added studies.
On the standard tasks, several added models outperform the best copy baseline by more than 5%: WFR-FM and DeepRUOT reduce error by 9% on day-4 interpolation, while scDiffEq and JKOnet* reduce error by 20% and 9% on day-6 forecasting. However, none of the models improves on the copy baseline by more than 5% on the three held-out fates. scDiffEq comes closest, with 2% lower error on neutrophils.
Across the added models, SF²M, Metric FM, MMFM, WFR-FM, scDiffEq, JKOnet*, and TIGON address all 32 held-out tasks with three seeds. DeepRUOT supports six Cuomo and Hagai tasks. We will include results for each seed, standard deviations, model adaptations, and fixed computational budgets in the appendix.
W2/Q3. Dataset diversity and biological coverage
Our original submission included seven time-course datasets covering embryoid bodies and five mouse datasets. We tested generalization to unseen biological groups across LARRY, CellTag, and Pijuan-Sala. However, LARRY provided most of our controlled analyses, so further testing across more diverse datasets is needed to determine how broadly these findings apply. To address this, we have added five more studies, including EMT, each with one biological group held out from training.
For each task, we average each model's RMSE across seeds and divide it by the best copy-baseline RMSE. The table reports the median ratio across tasks for each study, with the range in parentheses. A ratio below 1 means the model performs better than the copy baseline. To technically clarify, each task contributes to the ratio equally, while the best model can differ between tasks. The ranges show differences across each study’s tasks, not confidence intervals thereof. Also, "Lower" and "higher" mean more than 5% better or worse (not statistically significant differences).
| Added study | Held-out biology | Tasks | Median model/copy ratio (range) | Versus best copy baseline |
|---|---|---|---|---|
| Cook EMT | Cancer cell line | 9 | 5.31× (2.65–7.23×) | Higher on all 9 |
| Loh embryoid bodies | Genotype | 4 | 0.93× (0.88–1.00×) | Lower on 3; within 5% on 1 |
| Cuomo endoderm | Donor | 4 | 1.08× (1.03–1.19×) | Within 5% on 2; higher on 2 |
| Hagai immune response | Stimulus | 2 | 2.28× (1.99–2.56×) | Higher on both |
| McFarland drug response | Cancer cell line | 13 | 2.33× (1.61–3.50×) | Higher on all 13 |
The table excludes incomplete runs, pretrained models, and models or preprocessing methods that use target-time cells (i.e. day 6 cells) during training. As in LARRY, we include PRESCIENT, which for validation selects the best checkpoint using target-time cells from non-held-out fates (response to Reviewer Qcnt, Q3). We did not run PRESCIENT on Hagai or Cuomo because of computational cost.
On Loh, PRESCIENT, starting from the held-out genotype's own cells, reduces error by 6.1–11.8% on three genotypes compared with the best copy baseline. scDiffEq also reduces error by 10.3% and 7.7% on two genotypes. Without PRESCIENT, the Loh median ratio is 0.95× (0.90–1.03×), with improvements on two tasks. Cook and McFarland have median ratios of 5.41× and 2.34×, respectively.
To test whether the inability for cross-fate generalization can be attributed to lack of the held-out fate cells in the training set, we compared each model on identical target cells with the held-out fate excluded from training versus returned to training at a matched training size (see 6AEn, W2; Qcnt, W2/Q2). We applied the nine configurations of the LARRY control to every task (six on Hagai and Cuomo, where PRESCIENT has no scored results). We evaluated where the error is more than 5% higher when the held-out fate was excluded in the training set than included, with the condition that the three-seed ranges were non-overlapping. This was consistent in 65 of 81 comparisons on Cook, 9 of 12 on Hagai, 10 of 36 on Loh, and 0 of 24 on Cuomo. McFarland has no matched control, so its higher errors than copying cannot be attributed to withholding the group.
The results suggest models perform worse than the copy baseline on the EMT cell lines, drug-response cell lines, and immune-response conditions. The control experiments also link holding out a biological group to higher errors on Cook and Hagai. However, results on Loh and Cuomo are mixed, and holding out a group has a smaller effect on these studies.
References
[1] Atanackovic, L., et al. (2025). Meta flow matching: Integrating vector fields on the Wasserstein manifold. International Conference on Learning Representations (ICLR).
Response to Reviewer VwXD
We thank the reviewer for the detailed feedback and for recognizing the value of our benchmark in evaluating models across different tasks and perspectives. We have built on this framework by evaluating the six suggested models, adding five new studies, and further investigating why models struggle to predict unseen populations. We address each point below.
W1/Q1. Why models fail, and future guidance.
We investigate why models struggle to predict unseen cell populations, using LARRY as a case study. Our analyses suggest two possible reasons: (1) gene expression latent representations may remove information about populations not seen during training, and (2) some widely adopted latent representation methods may fail to capture biologically meaningful relationships among genes. These results describe patterns we observe in the models; further, a (3) biological explanation is proposed below.
Context: Models fail to identify unseen cell fates. In the below table, we demonstrate that even with all genes available, models struggle to identify the correct fate from earlier cells, increasing forecasting difficulty for even later cells. To note, these are not strict forecasting tasks because day-6 cells from other populations remain in training. For the sake of analysis, clones are grouped by their descendants' eventual fate, even if their earlier cells have not yet developed that identity.
The proportion of day-6 cells with fate correctly identified. Actual cells use dataset annotations, while predicted cells are labeled via a 15-nearest-neighbor classifier, trained relative to ground truth fate labels. Maximum values are calculated across eleven fitted models, after averaging stochastic configurations over seeds. Actual proportions are below 100% because clones are assigned the fate of most of their day-6 descendants. However, some clones also produce other cell types, and some descendants are still undifferentiated at day 6.
Across all three tasks, models predict a lesser proportion of cells of the actual final fate than the copy baseline. For example, the actual day-6 population comprises 95% neutrophils, compared with 55% in the copied cells and at most 47% in model predictions. This shows that models struggle to capture the developmental changes that distinguish different cell fates, even when population-average expression scores may not fully reflect this error.
Latent representation can distort held-out population information. Several evaluated implementations, including PRESCIENT, OT-CFM, SF²M, Metric FM, DeepRUOT, WFR-FM, and MMFM, compress gene expression using training-fitted PCA. This representation may poorly preserve expression patterns important to a population absent from training. To examine this possibility, we compared a raw-copy control with PCA-reconstructed copies of the same held-out basophil clones’ day-4 cells. The raw-copy control leaves their original gene expression unchanged. The reconstructed controls pass those cells through 50 principal components fitted on either a training set excluding basophils or a size-matched set including separate basophil training clones. No dynamics model is applied: all three outputs are evaluated directly against the same day-6 population. The raw copy achieves RMSE 0.0686, compared with 0.1309 after reconstruction through the basophil-excluded PCA and 0.0728 through the basophil-included PCA—increases of 91% and 6%, respectively. This fixed-seed diagnostic shows that the representation pipeline can worsen the starting cells’ agreement with the future population, particularly when the fate is absent from PCA fitting. Future work should examine how different representations preserve gene-expression information from held-out populations, since this can limit downstream predictions, especially as exhaustively incorporating every possible developmental fate across biological contexts into a latent representation is experimentally impractical.