- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes scLiveEval, a dynamic and contamination-aware benchmark for evaluating single-cell foundation models. The framework ingests newly released public datasets, standardizes the data, and screens for potential pretraining data leakage using release dates and provenance metadata. Several recent foundation models and standard baselines are evaluated zero-shot on novel cell-type detection, cross-platform integration, and human-model annotation disagreement. The results show that model rankings depend heavily on the biological and technical context, and simple baselines remain quite competitive.
Critical strengths
-
The motivation is timely. Pretraining data leakage is a growing issue in single-cell foundation models. Combining temporal cutoffs with provenance tracking is a sensible design that could become useful community infrastructure if properly maintained.
-
The empirical coverage is broad and up to date, comparing a large group of recent foundation models against classical methods like PCA and scVI on newly released data.
Critical weaknesses
-
The core claims of a living, agent-orchestrated, and contamination-aware benchmark are largely unsubstantiated. Despite the framing, the paper only presents a single static evaluation snapshot rather than tracking model rankings across updates over time. Looking at the supplementary code, the workflow is just a collection of standard dataset-specific preprocessing scripts rather than an autonomous agent, with no evaluation of pipeline reliability or manual effort required. Moreover, while contamination screening files are included in the code config, the paper never analyzes the audit results in the text or tests whether potential exposure actually inflates downstream performance.
-
The cross-platform evaluation is heavily confounded. As noted in the supplement, the platform cohorts differ simultaneously in donor composition, gene panels, label granularity, and batch definitions. Because targeted gene panels interact differently with each model's vocabulary, ranking shifts cannot be cleanly attributed to the sequencing technology itself. There are also minor numerical inconsistencies between the main text and the supplement regarding the cross-generation rank correlations.
-
The task designs and statistical support are relatively weak. Novel-cell detection relies on single splits with very small query sizes for some cohorts and no confidence intervals. Several models perform worse than random guessing, and relying on a single fixed nearest-neighbor heuristic makes it unclear whether the results reflect representation quality or the scoring rule. Similarly, the annotation disagreement analysis is limited to a single two-donor dataset validated only by literature markers rather than independent ground truth.
Questions for authors
-
Based on your contamination audit files, what is the actual breakdown of risk categories, and does potential pretraining exposure noticeably change model performance on any task?
-
Given that the supplementary code consists of standard retrieval and dataset-specific processing scripts, could you clarify what makes this an agent-orchestrated system and how much manual intervention is needed when ingesting a new dataset?
-
Can you provide confidence intervals for the novel-cell detection metrics, and explain why several models yield scores well below random chance?
-
In the cross-platform tasks, could you restrict the evaluation to a shared gene subset to separate platform effects from vocabulary mismatch?
4 comments
Robustness Analyses
Q3:
Thank you for the suggestion. We calculated 95% bootstrap confidence intervals using 2,000 resamples across all six cohorts and 18 representations, keeping the resampling indices, reference sets, data splits, and representations consistent. We also checked donor-cluster sensitivity where donor info was available.
Our re-evaluation reproduced the original AUROC and AP results for all 108 cohort–representation combinations, confirming that the score directions followed our evaluation setup. An AUROC below chance means that the model's fixed scores ranked seen and novel cells in the exact reverse of the expected order.
For instance, under scGPT, all 54 mechanoreceptor queries in LA-2 had neuroblast nearest-reference neighbors, resulting in an AUROC of 0.021. Out of these, 51 queries came from a single donor, which limits generalizability across donors. For the full LA-2 query set, scGPT achieved an AUROC of 0.184, with a cell-bootstrap interval of [0.121, 0.256] and a donor-cluster interval of [0.091, 0.512].
We will add these confidence intervals and diagnostics to the Supplementary Material and briefly clarify them in the Methods and Results. We will keep the original score directions and note that the optimized F1 score does not have a universal random-chance baseline of 0.5.
Q4:
Thank you for this suggestion. We added a shared-gene control, which showed that the overall model rankings were largely preserved after harmonizing input genes across platforms.
Specifically, we restricted the 10x GEM-X Flex v2, Parse Evercode V4, and ScaleBio QuantumScale datasets to the same 17,928 genes, with consistent Ensembl identifiers and gene symbols. All 18 representations were evaluated using these shared inputs, while preserving cell identities and retained expression values.
We evaluated BioScore and Batch ASW and compared the overall model rankings between the original and shared-gene conditions within each platform. Spearman correlations were 0.969 for Flex, 0.876 for Parse, and 0.960 for QuantumScale, indicating strong agreement across all three datasets. Although some individual models changed rank, the overall ranking patterns remained highly consistent. These results support the robustness of our main comparative findings to differences in input-gene coverage.
We will include the methods, results, and performance and ranking plots in Supplementary Section 6.5 (“Shared-gene control across platform cohorts”), Figures S16–S17.
Thank you again for your insightful comments and the time you have dedicated to reviewing our manuscript. We wanted to gently follow up on our previous response to check if our revisions and clarifications have adequately resolved your concerns. If there are any remaining questions or areas that require further explanation, please do not hesitate to let us know. We are very eager to ensure the manuscript meets your expectations.
Thank you for the additional experiments. They address several of my original concerns. A publicly accessible leaderboard with a verifiable update history should be part of this paper’s contribution. The authors provide a local demo and plan to launch the site after publication. That leaves the paper’s central claim of a continuously updated benchmark unsubstantiated. Almost all the figures look like rough AI-generated drafts. For example, the text are much smaller than the body text, and the crowded layouts make several. Taken together, these issues leave me with the impression that the work is still unfinished, and I do not think the current version meets the bar for acceptance at ICLR. I considered lowering my score, but given the effort the authors put into the response and the additional experiments, I am keeping my original score.
Dynamic Evaluation, Workflow Automation
Thank you for highlighting this gap. We have now completed 144 evaluations of 15 scFMs and 3 baselines across eight PBMC datasets spanning 2019–2026. Using a combined score for biological conservation and batch mixing, five different methods ranked first across the eight datasets ecifically:
These findings reveal both consistently top-performing models and dataset-dependent shifts in leadership. They also highlight why conclusions from a single benchmark snapshot should be reassessed as new cohorts and technologies become available. We will incorporate these results into Section 5.2, clearly framing this study as a completed retrospective evaluation, while discussing the prospective operation of a continuously updated platform separately.
Q1:
Our updated date check covers 165 dataset–model pairs (11 dataset versions × 15 pretrained models): 132 are classified as no leakage, 31 as unresolved, and 2 as potential leakage. "Unresolved" means the required chronological order could not be established from the dates alone. For example, ScaleBio v1.1–CellVQ and Flex v1–OmniCell remain unresolved. Because the official materials do not provide complete training dataset lists for the model versions we tested, we cannot fully check whether our evaluation data were included in pretraining. The two flagged pairs are LA-1–scFoundation and ScaleBio v1.1–TranscriptFormer, based on shared sample or source identifiers.
For novel-cell detection, we excluded LA-1 across all 18 representations and recalculated cohort averages. Mean AUROC dropped for 17 of the 18 methods (including all three baselines), with decreases of 0.161 for scGPT and 0.125 for scFoundation. However, the top-performing method for each metric remained unchanged. Spearman correlations between rankings before and after exclusion were 0.924 for AUROC, 0.963 for AUPRC, and 0.880 for best F1.
For platform comparisons, we excluded TranscriptFormer from both generations in both platform families, leaving 17 representations. BioScore rank agreement remained higher for ScaleBio than Flex (0.868 versus 0.515). These checks confirm that our main rankings and cross-platform conclusions remain stable when these data are excluded.
The methods and results are detailed in Supplementary Material, Section 3.2 (“Overlap screening and exclusion sensitivity”). The updated table (
datet_model_date_check.xlsx) is included underexposure_risk/date_screen_20261112/inScLiveEval_supplement_code.zipAnd we will upload the files soon.Q2:
Thank you for asking us to clarify how the agent operates and where human input is involved. In our workflow, an LLM-based controller reads the evaluation request, selects the appropriate skills, runs computational tools, and checks the outputs to decide the next step. The skills define the procedures and rules for each task, while scripts handle tasks like preprocessing, embedding generation, and evaluation. The controller coordinates these steps based on the user's request and the current execution state, and the reusable skills and tool interfaces are designed to work across different agent runtimes.
To test this, in a case study using a 2019 PBMC dataset, we gave Codex a simple prompt: “Please find an annotated PBMC single-cell transcriptomic dataset from 2019 and evaluate model performance.” The agent automatically downloaded the dataset from CELLxGENE, preprocessed it, generated model embeddings, checked cell ID alignment, computed scIB metrics, and generated UMAP plots. In this scenario, human input was limited to the initial high-level request and the analysis protocol, demonstrating a high level of automation across data acquisition, processing, evaluation, and visualization.