acceptodds
Under review as a conference paper at ICLR 2027

BEYOND STATIC BENCHMARKS: LIVING EVALUATION FOR SINGLE-CELL FOUNDATION MODELS

Abstract

Single-cell datasets and sequencing technologies are evolving rapidly, yet the evaluation of single-cell foundation models (scFMs) still relies largely on fixed benchmarks that may overlap with expanding pretraining corpora. We introduce scLiveEval, a living, contamination-aware framework that continuously incorporates newly released datasets and updates scFM evaluation over time while screening for potential data exposure using available temporal and provenance evidence. Study-defined novel cell populations are used to evaluate generalization to emerging biology; comparisons across platform-specific cohorts assess whether relative model rankings changes across measurement contexts; and, as newly annotated datasets enter the benchmark, human–model annotation disagreement analysis examines divergence between model predictions and supplied labels, highlighting populations that require further review and informing model applicability. We observe clear context dependence in model behavior: novel-population detectability varies across datasets, while biological preservation, batch integration, and model rankings differ across platform-specific cohorts. Cross-model consensus also reveals recurrent annotation disagreements supported by donor-specific transcriptional evidence. These findings motivate a shift from static benchmarking toward living evaluation as single-cell data and technologies continue to evolve.

Primary area:
datasets and benchmarks
Keywords:
Single-cell foundation models, Benchmarking, Distribution shift

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AYnpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes scLiveEval, a dynamic and contamination-aware benchmark for evaluating single-cell foundation models. The framework ingests newly released public datasets, standardizes the data, and screens for potential pretraining data leakage using release dates and provenance metadata. Several recent foundation models and standard baselines are evaluated zero-shot on novel cell-type detection, cross-platform integration, and human-model annotation disagreement. The results show that model rankings depend heavily on the biological and technical context, and simple baselines remain quite competitive.

Critical strengths
  1. The motivation is timely. Pretraining data leakage is a growing issue in single-cell foundation models. Combining temporal cutoffs with provenance tracking is a sensible design that could become useful community infrastructure if properly maintained.

  2. The empirical coverage is broad and up to date, comparing a large group of recent foundation models against classical methods like PCA and scVI on newly released data.

Critical weaknesses
  1. The core claims of a living, agent-orchestrated, and contamination-aware benchmark are largely unsubstantiated. Despite the framing, the paper only presents a single static evaluation snapshot rather than tracking model rankings across updates over time. Looking at the supplementary code, the workflow is just a collection of standard dataset-specific preprocessing scripts rather than an autonomous agent, with no evaluation of pipeline reliability or manual effort required. Moreover, while contamination screening files are included in the code config, the paper never analyzes the audit results in the text or tests whether potential exposure actually inflates downstream performance.

  2. The cross-platform evaluation is heavily confounded. As noted in the supplement, the platform cohorts differ simultaneously in donor composition, gene panels, label granularity, and batch definitions. Because targeted gene panels interact differently with each model's vocabulary, ranking shifts cannot be cleanly attributed to the sequencing technology itself. There are also minor numerical inconsistencies between the main text and the supplement regarding the cross-generation rank correlations.

  3. The task designs and statistical support are relatively weak. Novel-cell detection relies on single splits with very small query sizes for some cohorts and no confidence intervals. Several models perform worse than random guessing, and relying on a single fixed nearest-neighbor heuristic makes it unclear whether the results reflect representation quality or the scoring rule. Similarly, the annotation disagreement analysis is limited to a single two-donor dataset validated only by literature markers rather than independent ground truth.

Questions for authors
  1. Based on your contamination audit files, what is the actual breakdown of risk categories, and does potential pretraining exposure noticeably change model performance on any task?

  2. Given that the supplementary code consists of standard retrieval and dataset-specific processing scripts, could you clarify what makes this an agent-orchestrated system and how much manual intervention is needed when ingesting a new dataset?

  3. Can you provide confidence intervals for the novel-cell detection metrics, and explain why several models yield scores well below random chance?

  4. In the cross-platform tasks, could you restrict the evaluation to a shared gene subset to separate platform effects from vocabulary mismatch?

4 comments
AuthorsOpenReview ↗

Dynamic Evaluation, Workflow Automation

Thank you for highlighting this gap. We have now completed 144 evaluations of 15 scFMs and 3 baselines across eight PBMC datasets spanning 2019–2026. Using a combined score for biological conservation and batch mixing, five different methods ranked first across the eight datasets ecifically:

  • UCE achieved the best mean rank (2.625) and remained among the top five on every dataset.
  • SCimilarity followed with a mean rank of 3.125, and both methods placed in the top three on six datasets.
  • scCello and scFoundation each secured first place on two datasets.
  • scVI + Harmony ranked first on the 2023 dataset and entered the top five on six datasets, demonstrating that conventional baselines remain competitive.

These findings reveal both consistently top-performing models and dataset-dependent shifts in leadership. They also highlight why conclusions from a single benchmark snapshot should be reassessed as new cohorts and technologies become available. We will incorporate these results into Section 5.2, clearly framing this study as a completed retrospective evaluation, while discussing the prospective operation of a continuously updated platform separately.


Q1:

Our updated date check covers 165 dataset–model pairs (11 dataset versions × 15 pretrained models): 132 are classified as no leakage, 31 as unresolved, and 2 as potential leakage. "Unresolved" means the required chronological order could not be established from the dates alone. For example, ScaleBio v1.1–CellVQ and Flex v1–OmniCell remain unresolved. Because the official materials do not provide complete training dataset lists for the model versions we tested, we cannot fully check whether our evaluation data were included in pretraining. The two flagged pairs are LA-1–scFoundation and ScaleBio v1.1–TranscriptFormer, based on shared sample or source identifiers.

For novel-cell detection, we excluded LA-1 across all 18 representations and recalculated cohort averages. Mean AUROC dropped for 17 of the 18 methods (including all three baselines), with decreases of 0.161 for scGPT and 0.125 for scFoundation. However, the top-performing method for each metric remained unchanged. Spearman correlations between rankings before and after exclusion were 0.924 for AUROC, 0.963 for AUPRC, and 0.880 for best F1.

For platform comparisons, we excluded TranscriptFormer from both generations in both platform families, leaving 17 representations. BioScore rank agreement remained higher for ScaleBio than Flex (0.868 versus 0.515). These checks confirm that our main rankings and cross-platform conclusions remain stable when these data are excluded.

The methods and results are detailed in Supplementary Material, Section 3.2 (“Overlap screening and exclusion sensitivity”). The updated table (datet_model_date_check.xlsx) is included under exposure_risk/date_screen_20261112/ in ScLiveEval_supplement_code.zip And we will upload the files soon.


Q2:

Thank you for asking us to clarify how the agent operates and where human input is involved. In our workflow, an LLM-based controller reads the evaluation request, selects the appropriate skills, runs computational tools, and checks the outputs to decide the next step. The skills define the procedures and rules for each task, while scripts handle tasks like preprocessing, embedding generation, and evaluation. The controller coordinates these steps based on the user's request and the current execution state, and the reusable skills and tool interfaces are designed to work across different agent runtimes.

To test this, in a case study using a 2019 PBMC dataset, we gave Codex a simple prompt: “Please find an annotated PBMC single-cell transcriptomic dataset from 2019 and evaluate model performance.” The agent automatically downloaded the dataset from CELLxGENE, preprocessed it, generated model embeddings, checked cell ID alignment, computed scIB metrics, and generated UMAP plots. In this scenario, human input was limited to the initial high-level request and the analysis protocol, demonstrating a high level of automation across data acquisition, processing, evaluation, and visualization.

AuthorsOpenReview ↗

Robustness Analyses

Q3:

Thank you for the suggestion. We calculated 95% bootstrap confidence intervals using 2,000 resamples across all six cohorts and 18 representations, keeping the resampling indices, reference sets, data splits, and representations consistent. We also checked donor-cluster sensitivity where donor info was available.

Our re-evaluation reproduced the original AUROC and AP results for all 108 cohort–representation combinations, confirming that the score directions followed our evaluation setup. An AUROC below chance means that the model's fixed scores ranked seen and novel cells in the exact reverse of the expected order.

For instance, under scGPT, all 54 mechanoreceptor queries in LA-2 had neuroblast nearest-reference neighbors, resulting in an AUROC of 0.021. Out of these, 51 queries came from a single donor, which limits generalizability across donors. For the full LA-2 query set, scGPT achieved an AUROC of 0.184, with a cell-bootstrap interval of [0.121, 0.256] and a donor-cluster interval of [0.091, 0.512].

We will add these confidence intervals and diagnostics to the Supplementary Material and briefly clarify them in the Methods and Results. We will keep the original score directions and note that the optimized F1 score does not have a universal random-chance baseline of 0.5.


Q4:

Thank you for this suggestion. We added a shared-gene control, which showed that the overall model rankings were largely preserved after harmonizing input genes across platforms.

Specifically, we restricted the 10x GEM-X Flex v2, Parse Evercode V4, and ScaleBio QuantumScale datasets to the same 17,928 genes, with consistent Ensembl identifiers and gene symbols. All 18 representations were evaluated using these shared inputs, while preserving cell identities and retained expression values.

We evaluated BioScore and Batch ASW and compared the overall model rankings between the original and shared-gene conditions within each platform. Spearman correlations were 0.969 for Flex, 0.876 for Parse, and 0.960 for QuantumScale, indicating strong agreement across all three datasets. Although some individual models changed rank, the overall ranking patterns remained highly consistent. These results support the robustness of our main comparative findings to differences in input-gene coverage.

We will include the methods, results, and performance and ranking plots in Supplementary Section 6.5 (“Shared-gene control across platform cohorts”), Figures S16–S17.

Thank you again for your insightful comments and the time you have dedicated to reviewing our manuscript. We wanted to gently follow up on our previous response to check if our revisions and clarifications have adequately resolved your concerns. If there are any remaining questions or areas that require further explanation, please do not hesitate to let us know. We are very eager to ensure the manuscript meets your expectations.

Thank you for the additional experiments. They address several of my original concerns. A publicly accessible leaderboard with a verifiable update history should be part of this paper’s contribution. The authors provide a local demo and plan to launch the site after publication. That leaves the paper’s central claim of a continuously updated benchmark unsubstantiated. Almost all the figures look like rough AI-generated drafts. For example, the text are much smaller than the body text, and the crowded layouts make several. Taken together, these issues leave me with the impression that the work is still unfinished, and I do not think the current version meets the bar for acceptance at ICLR. I considered lowering my score, but given the effort the authors put into the response and the additional experiments, I am keeping my original score.

Reviewer_aC3COpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents scLiveEval, an updating evaluation framework for single-cell foundation models. It combines dataset provenance and training-cutoff checks with tests of held-out cell populations, platform-specific performance, and disagreements with supplied annotations. The experiments show that model rankings depend on the dataset and measurement context. A donor-aware analysis also identifies annotation disagreements with supporting marker evidence. The main contribution is an evaluation workflow intended to keep these comparisons useful as new data and models appear.

Critical strengths
  1. The motivation is clear. The paper connects expanding pretraining datasets with new biological populations and sequencing technologies, then builds evaluation tasks around these changes. The comparisons with PCA and scVI make the results useful for choosing representations.

  2. The annotation analysis goes beyond counting disagreements. It checks evidence across donors and separates marker-supported cases, technical artifacts, and unresolved cases. This makes the examples more convincing than model agreement alone.

Critical weaknesses
  1. The central living-evaluation claim is not demonstrated in the main results. Section 3.1 reports a completed exposure audit, but the results do not show its classifications or their effect on the reported comparisons. There are also no successive benchmark snapshots. Differences between datasets or platform generations do not establish that the updating workflow works. The added value over an expanded static benchmark remains unclear.

  2. Section 5.4 does not establish what foundation-model consensus adds to annotation review. The reported analysis votes over foundation models only, although the earlier experiments show that simple representations can be competitive. Marker-supported examples are useful, but without a comparison to ordinary neighbor transfer or marker-based screening, they do not show why the proposed consensus is needed.

Questions for authors
  1. Can you point to completed benchmark updates and summarize what changed between versions? For the results already shown, which dataset-model pairs were low risk, flagged, or unknown, and were any pairs excluded?

  2. Were the reported annotation candidates also flagged by PCA or scVI neighbor transfer? If this was checked, please summarize the overlap and any cases that specifically benefited from foundation-model consensus.

7 comments
AuthorsOpenReview ↗

Completed Benchmark Updates and Contamination Assessment

Q1:

Completed Benchmark Updates

Yes. We have expanded our benchmark using the original 2019 PBMC dataset and seven additional annotated datasets spanning 2020–2026. Compared to the static evaluation in our initial submission, this update covers cross-year comparisons across the same 15 single-cell foundation models (scFMs) and 3 baselines, completing all 144 model–dataset evaluations. For each cohort, we generated model embeddings, calculated scIB metrics, and computed overall rankings using the formula:

Overall=0.6×AvgBIO+0.4×AvgBATCH.\mathrm{Overall} = 0.6 \times \mathrm{AvgBIO} + 0.4 \times \mathrm{AvgBATCH}.

The heatmap identifies UCE, SCimilarity, scCello, and scFoundation as the four best-performing methods by mean rank, with average ranks of 2.625, 3.125, 3.625, and 4.250, respectively. UCE remained within the top five on all eight datasets, and both UCE and SCimilarity ranked among the top three on six datasets. scCello and scFoundation also performed strongly, although their positions varied across cohorts.

The baselines remained competitive, particularly scVI + Harmony, which achieved a mean rank of 5.250 and entered the top five on six datasets, outperforming many foundation models. scVI occupied a middle-ranking position overall, with a mean rank of 9.250, while PCA ranked lower, with a mean rank of 15.875. These results show that foundation models do not uniformly outperform conventional baselines and that their relative advantages depend on the evaluation dataset.

Together, these findings identify consistently competitive methods while revealing dataset-dependent ranking changes. They support repeated evaluation as new datasets become available, allowing earlier conclusions about model performance to be reassessed across different cohorts and technologies.

We will present the completed rankings and heatmap in Section 5.2.


Contamination Audit and Sensitivity Analyses

  • 132 N (“no leakage”): The dataset release dates fall after the recorded model dates. This assumes that the dataset dates reflect when the underlying data first became public, that the model dates apply to the specific checkpoints we evaluated, and that pretraining used publicly available data.

  • 31 unresolved: The dates alone do not establish temporal separation. For pairs such as ScaleBio v1.1–CellVQ and Flex v1–OmniCell, incomplete training documentation also limits further verification. The official materials we reviewed do not provide complete training-dataset lists for the model versions we tested, so we cannot fully check whether our evaluation data were included in pretraining.

  • 2 P (“potential leakage”): LA-1–scFoundation and ScaleBio v1.1–TranscriptFormer, flagged through shared sample or source identifiers.

We assessed targeted exclusions separately through two sensitivity analyses using existing evaluation scores.

For novel-cell detection, we excluded LA-1 across all 18 representations. The leading method for each metric remained unchanged. Before-versus-after Spearman rank correlations were 0.924 for AUROC, 0.963 for AUPRC, and 0.880 for best F1.

For platform-generation comparisons, we checked how much the conclusions depended on TranscriptFormer by excluding it from both generations and retaining the same 17 representations in both platform families. BioScore rank agreement remained higher for ScaleBio than Flex (0.868 versus 0.515).

These results show that the main ranking patterns and cross-platform conclusions were preserved under the targeted exclusions. Some absolute scores changed after cohort exclusion.

The full procedures and results are reported in Supplementary Material, Section 3.2. The complete pair-level classifications are provided in datet_model_date_check.xlsx within ScLiveEval_supplement_code.zip.

AuthorsOpenReview ↗

Baseline Comparisons for Annotation Consensus

Q2:

PCA and scVI neighbor transfer independently recovered all five candidate directions identified by the 15-model FM consensus. The mean direction-wise cell-level Jaccard was 0.633 for PCA and 0.558 for scVI, indicating that the direction-level findings were not unique to the FM vote. Because the original framework was deliberately designed around FM-based consensus, PCA and scVI were not included in the primary analysis; their inclusion here therefore serves as an independent reinforcement check rather than a replacement for the FM consensus.

At the cell level, of the 448 FM-consensus candidate cells, 272 were assigned the same alternative by both baselines, 78 by PCA only, 20 by scVI only, and 78 by neither. The 78 cells flagged by neither baseline were concentrated in:

  • Int/Mem B → Naive B: 54 cells.
  • CD4+ T → Naive CD8+ T: 20 cells.
  • Classical Monocytes → CD16 Monocytes: 4 cells.

These represent the cell-level examples where the FM consensus adds review value beyond ordinary neighbor transfer. Thus, the added value of the FM consensus provides a framework for prioritizing annotation disagreements and contributes a small number of cell-level candidates not captured by single-representation baseline.

We will update the experiment in supplementary.

I appreciate the authors' response. After reading this, I still have a few more questions:

  1. The idea of building a living benchmark is interesting. However, I think if a benchmark is really called living, which should have a frequently updating leaderboard which keeps evaluating the performance of a new model by measuring it on unseen data. Otherwise, when the results are out, the benchmark becomes static again. There will be no real so call "living". Do the authors have any plan on building this platform?

  2. To comprehensively evaluate scFMs, I think more tasks should be included. Especially, perturbation respose modeling is a key task to reflect single-cell modeling capability. Is there a reason why the authors fouced on the three tasks discussed in the paper?

Reviewer_xSJNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SCLIVEEVAL, a living and contamination-aware evaluation framework for single-cell foundation models (scFMs). The authors argue that conventional benchmarks based on fixed public datasets become increasingly unreliable as model pretraining corpora expand. Evaluation datasets may overlap with pretraining data, and static benchmarks cannot adequately reflect newly discovered cell populations, emerging sequencing platforms, or evolving annotation requirements.

Critical strengths

Rather than relying on a single downstream metric, the paper evaluates biological distribution shifts, sequencing-platform shifts, biological structure preservation, batch integration, and annotation uncertainty. This multidimensional design better reflects how scFMs are used in practice.The study compares 15 pretrained foundation models and three data-fitted baselines across multiple tissues, novel or low-abundance populations, and three emerging sequencing platforms. The experiments provide useful evidence that scFM performance is strongly context-dependent.

Critical weaknesses

The contamination analysis is useful as an initial risk check, but its conclusions are necessarily limited by the available metadata. Release dates indicate only whether exposure was possible, not whether a dataset was actually used for pretraining. Likewise, accession numbers and file hashes may miss substantial overlap when the same data have been reprocessed, reformatted, or deposited through different repositories. This issue is difficult to resolve because several models do not provide complete records of their pretraining data. The resulting labels should therefore be interpreted as indicators of possible exposure rather than definitive evidence for or against contamination.

Questions for authors

How should readers interpret the contamination-risk labels when complete pretraining dataset lists are unavailable? In particular, can the framework detect overlap when the same underlying cells have been reprocessed, renamed, or deposited in different repositories?In the platform analysis, how do the authors separate the effect of sequencing technology from differences in donor composition, sample preparation, gene coverage, and batch structure? Would it be possible to validate the conclusions using matched samples measured on multiple platforms?

4 comments
AuthorsOpenReview ↗

Data Overlap Screening and Platform Robustness

Q1:

Thank you for highlighting this distinction. We re-checked 165 dataset–model pairs across 11 dataset versions and 15 pretrained models.

The updated categories are: 132 N (“no leakage”): The dataset release dates fall after the model dates. This assumes that the dataset dates reflect when the data first became public, that the model dates apply to the specific checkpoints we evaluated, and that pretraining relied on publicly available data. 31 unresolved: The dates alone don't establish clear temporal separation, and incomplete training documentation from the model creators prevents further verification. For example, ScaleBio v1.1–CellVQ and Flex v1–OmniCell remain unresolved. Because the official materials do not provide complete training dataset lists for the model versions we tested, we cannot fully check whether our evaluation data were included in pretraining. 2 P(“potential leakage”): LA-1–scFoundation and ScaleBio v1.1–TranscriptFormer, flagged due to shared sample or source identifiers.

We also ran two sensitivity analyses using our existing evaluation scores. For novel-cell detection, excluding LA-1 across all 18 representations didn't change the top-performing method for any metric. The before-and-after Spearman rank correlations were 0.924 for AUROC, 0.963 for AUPRC, and 0.880 for best F1. For platform comparisons, we looked at how much results depended on TranscriptFormer by dropping it from both generations, leaving 17 representations in both platform families. BioScore rank agreement remained higher for ScaleBio than Flex (0.868 versus 0.515), and cross-family rank changes stayed weakly correlated (ρ = 0.136, p = 0.603). These checks confirm that our main rankings and cross-platform conclusions are robust to these targeted exclusions, even though individual absolute scores changed when cohorts were removed. The full procedures and results are in Supplementary Material, Section 3.2, and the complete classifications are provided in datet_model_date_check.xlsx within the supplementary code package.

Q2:

Thank you for highlighting this point. We have optimized the format_adapter skill and its underlying script to strengthen overlap detection across datasets available for comparison through two complementary mechanisms.

First, instead of using dataset-level mean-expression fingerprints, we now use per-cell raw-count fingerprints. Starting with verified raw counts, the script standardizes gene names, gene order, and number formats, then computes an MD5 fingerprint from each cell’s gene-count vector and the standard gene set (ignoring file and cell names). Matching fingerprints allow us to identify identical count profiles even if files or cells have been renamed, rows or columns reordered, or data deposited in a different repository. This per-cell matching also catches partial overlap when datasets are subsetted, split, or merged, as long as the raw counts and canonical gene set remain unchanged.

Second, the formatting script now keeps basic sample information (such as study and sample IDs) along with the original sequencing library IDs and cell barcodes whenever available. Barcodes are matched within their original libraries. While shared study or sample accessions provide initial clues for source-level reuse, cell-level matching via original libraries and barcodes provides definitive verification. By systematically leveraging these tracking elements, our framework reliably identifies reprocessed cells across diverse repositories.

AuthorsOpenReview ↗

Platform Robustness

Q3:

Thank you for this suggestion. Our original platform analysis already controlled for cell-type composition by evaluating all three platforms on the same six harmonized cell types. Specifically, BioScore was calculated across 20 balanced resamples (150 cells per type), giving us a consistent biological baseline to compare model performance. We also checked gene coverage using a shared set of 17,928 common genes and evaluated batch mixing using Batch ASW based on platform-specific batch annotations.

To further test this, we added a composition-matched evaluation using the 10x GEM-X Flex v2, Parse Evercode V4, and ScaleBio QuantumScale datasets. For each cell type, we found the smallest cell count available across the three platforms and randomly sampled that exact number from each platform without replacement. This gave each platform an identical composition of 7,667 cells: 1,375 B cells, 3,452 T cells, 1,295 NK cells, 1,000 classical monocytes, 376 non-classical monocytes, and 169 dendritic cells. We then re-generated embeddings for all 18 methods and recalculated both BioScore and Batch ASW.

When ranking models by the average of BioScore and Batch ASW, the Spearman correlations between the original and composition-matched rankings were 0.7585 for Flex, 0.9401 for Parse, and 0.9396 for ScaleBio. These results show that within-platform ranking patterns are largely preserved,especially for Parse and ScaleBio, supporting the robustness of our findings under a more standardized evaluation.

This new experiment establishes matched cell compositions across the existing datasets. We completely agree that profiling matched aliquots from the same donors across different platforms would provide even better validation, particularly for isolating donor and sample-prep effects. We are adding these supplementary experiments to the supplementary materials soon.

Thank you again for your insightful comments and the time you have dedicated to reviewing our manuscript. We wanted to gently follow up on our previous response to check if our revisions and clarifications have adequately resolved your concerns. If there are any remaining questions or areas that require further explanation, please do not hesitate to let us know. We are very eager to ensure the manuscript meets your expectations.

Thank you again for your time and consideration. We understand that this is a busy review period. If you have had a chance to review our responses and revisions, we would be grateful for any brief feedback on whether any substantive concerns remain. We would be happy to provide further clarification where needed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Benchmark Updates and Additional Validation

We thank the three reviewers and the Area Chair for their constructive feedback. During the rebuttal period, we completed additional experiments addressing the main concerns about benchmark updates, automation, data contamination, and evaluation robustness. These additions strengthen scLiveEval’s contribution as a reusable framework for evaluating models under changing biological and technical conditions.

1. Dynamic Evaluation and Automation

We completed a retrospective evaluation of 15 scFMs and three baselines across eight PBMC datasets spanning 2019–2026, producing 144 model–dataset evaluations. UCE, SCimilarity, scCello, and scFoundation achieved the strongest mean ranks. UCE remained in the top five on every dataset, with a mean rank of 2.625, while scVI + Harmony entered the top five on six datasets. These findings reveal both consistently competitive methods and dataset-dependent ranking changes, supporting reassessment as new datasets become available.

Alongside this expansion, a completed case using the 2019 PBMC dataset demonstrated agent-orchestrated embedding generation, cell-ID alignment checks, scIB evaluation, and visualization. The LLM-based controller selects reusable skills, invokes tools, and checks their outputs. This case demonstrates automation of routine evaluation once the inputs and analysis protocol are specified.

2. Data Screening and Contamination Assessment

We updated the format_adapter skill and its underlying script to combine per-cell raw-count fingerprints with sample, original-library, and barcode information. This strengthens detection of renamed or duplicated data when count profiles are preserved, and supports tracing reprocessed cells when reliable provenance is available.

We also re-examined 165 dataset–model pairs, classifying 132 as “no leakage” under the date-screening assumptions, 31 as unresolved, and two as potential leakage. The temporal classifications depend on the recorded dates and assumptions about public-data pretraining; incomplete training manifests limit definitive membership checks.

We then tested whether the flagged cases affected our conclusions. Excluding LA-1 preserved the leading method for every novel-cell detection metric, with ranking correlations of 0.924 for AUROC, 0.963 for AUPRC, and 0.880 for best F1. Excluding TranscriptFormer preserved the higher platform-generation rank agreement for ScaleBio than Flex (0.868 versus 0.515). These analyses support the robustness of the principal comparisons to the targeted exclusions.

3. Platform Robustness Checks

We completed both controls across Flex, Parse, and ScaleBio, evaluating all 18 methods.

For the matched-composition control, each platform contained 7,667 cells with identical cell-type counts. Correlations between the original and controlled rankings were 0.7585, 0.9401, and 0.9396, respectively. Rankings were largely preserved, particularly for Parse and ScaleBio.

For the shared-gene control, all methods received inputs restricted to the same 17,928 genes. Ranking correlations were 0.969, 0.876, and 0.960, respectively, showing strong agreement across all three platforms.

Together, these experiments show that the broad comparative patterns remain stable after harmonizing cell-type composition and input-gene coverage, strengthening confidence in our platform analyses.

4. Novel-Cell Uncertainty and Diagnostics

We computed 95% bootstrap confidence intervals using 2,000 resamples across six cohorts and 18 representations. Recalculation reproduced the original AUROC and AP results for all 108 cohort–representation combinations. We also added donor-group sensitivity analyses and nearest-reference diagnostics to explain below-chance performance under the fixed scoring rule. These additions make the uncertainty explicit and support our finding that novel-cell detectability depends on the target population and evaluation context.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.