- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work addresses evaluation of single-cell generative models. Compared to previous works, which focus on specific aspects of evaluation such as global statistical similarity, classifier agreement, or metrics derived from specific biological priors, this work proposes a more comprehensive, multidimensional framework for assessing biological fidelity. The framework covers different flavors of fidelity (lexical, syntactic, semantic, population) covering gene expression, functional programs, cellular identity, and population heterogeneity. Evaluation across several generative models reveals distinct failure modes that may be overlooked by more aggregate metrics.
Critical strengths
- The problem setting is very relevant, as evaluating the biological fidelity of single-cell generative models is an increasingly important challenge. Several prior works address this setting, but existing metrics often capture different (and potentially complementary) aspects of fidelity. From this point of view, a comprehensive and integrative study can provide meaningful and novel insights.
- The framework is well-structured, decomposing evaluation into four dimensions providing a useful organization of complementary biological properties. In particular, the use of reference-calibrated metrics is sensible, recognizing that unusually high values (e.g., marker enrichment or expression magnitude) can themselves indicate unrealistic behavior.
- Comprehensive experimental characterization. The work spans different categories of models, biological metrics, and calibration controls.
Critical weaknesses
- Limited downstream validation. The study identifies different discrepancies between generated and real cells across tested models, which could already support further metric-supported model development. However, it is not clear how and whether these discrepancies translate into meaningful differences into downstream biological applications, such as differential expression or pathway inference. Such analysis would further strengthen the proposed framework.
- Limited comparison with existing approaches. Given that many metrics are based or inspired by existing techniques, with the main contribution being their organization into a unified framework, the authors should provide direct and explicit comparisons with existing approaches and fully clarify the additional insights derived using CellFidelity (i.e., their integration).
Questions for authors
See Weaknesses
6 comments
Response to R3W1: Limited downstream validation (Continued)
2. Synthetic-cell augmentation reveals model-dependent downstream utility.
To further examine whether the fidelity differences identified by CellFidelity translate into practical differences in downstream applications, we evaluated synthetic-cell augmentation for DE and pathway inference. Using 800 real cells as a fixed reference, we compared three settings: 400 real cells alone (R400), the same 400 real cells augmented with 400 generated cells (R400+G400), and 800 generated cells alone (G800). The evaluation covered three cell-type contrasts and 10 paired technical resamples (Figure 7A–D).
We observed substantial differences across generative models. Adding scDiffusion-generated cells consistently reduced DE log2 fold-change reconstruction error relative to R400 alone, whereas CFGen provided smaller improvements (Figure 7A–B). In contrast, augmentation with scVI- and C2S-generated cells increased this error. These differences are broadly consistent with our main CellFidelity evaluation, in which scDiffusion showed stronger preservation of expression and functional-program properties, while the other models exhibited different fidelity deficiencies.
Importantly, improvements in DE reconstruction did not consistently lead to improvements in pathway inference (Figure 7C–D). This further emphasizes that downstream utility is task-dependent: synthetic cells that help reconstruct gene-wise expression differences may not necessarily preserve the coordinated biological signals required for reliable pathway-level interpretation.
These findings complement the controlled corruption experiments by demonstrating that fidelity differences are relevant not only under artificially introduced defects but also when evaluating the practical use of cells generated by different models.
Together, these findings support the importance of multidimensional fidelity assessment. Controlled corruptions connect specific fidelity defects to interpretable analytical consequences, while generated-cell augmentation shows that practical utility depends on both the model and the downstream task. We therefore interpret these experiments as evidence of downstream analytical relevance and model-dependent utility, rather than as universal predictive relationships between individual metrics and downstream performance.
Response to R3Q2: Limited comparison with existing approaches
We thank the reviewer for this important suggestion. We agree that CellFidelity should be evaluated in direct comparison with existing approaches, particularly because several of its constituent metrics are adapted from or inspired by established techniques. The main contribution of CellFidelity is not to introduce entirely new metrics, but to organize complementary measurements into a biologically interpretable framework that can characterize different failure modes of generated single-cell data. To clarify this contribution, we conducted systematic controlled-corruption experiments comparing CellFidelity with conventional evaluation measures and examining the additional diagnostic insights provided by their integration.
1. Direct comparison with established readouts.
We constructed nine controlled-corruption families using real B/plasma, endothelial, and neuronal cells, covering four aspects of biological fidelity: gene inclusion and expression magnitude (top-K transcriptome truncation, frequent-to-rare gene replacement, and detection-rate-stratified expression scaling), inter-gene dependence (within-program coordination shuffling), correspondence between expression and cellular identity (cell-type label mismatch, canonical-marker dropout, and identity-program dropout), and population heterogeneity (reference-subpopulation depletion and gene-wise variance compression).
Each corruption was applied at predefined strengths, with matched controls where applicable. The original, unperturbed real cells served as the clean baseline; matched controls were additional corruption-specific comparisons. Rather than assuming that each intervention should affect only one dimension, we examined both its intended effects and broader consequences across all 27 originally submitted CellFidelity metrics.
We compared these responses with four established evaluation readouts: mean gene-wise Wasserstein-1 distance (W1), mean absolute error in gene-wise zero fractions (ZeroMAE), squared maximum mean discrepancy (MMD²) in a fixed reference-PCA space, and held-out logistic-regression AUROC for distinguishing clean from corrupted cells. These measures represent conventional gene-level and distributional evaluation approaches rather than complete reproductions of published benchmarking frameworks. MMD² is also included among the original CellFidelity metrics and is reused here as a stand-alone global baseline.
This comparison was designed to distinguish two questions: whether an evaluation measure detects a discrepancy, and whether the combined measurements identify the biological property affected by that discrepancy. The aligned results are summarized in Figure 8A, with representative diagnostic contrasts in Figure 8B–D.
2. Examples where CellFidelity provides diagnostic information beyond individual discrepancy measures.
Gene coordination can deteriorate despite preserved gene-wise marginals. Independent within-program shuffling permutes each selected gene across cells, preserving its marginal expression distribution while disrupting cross-gene coordination. Accordingly, gene-wise W1 and ZeroMAE remain approximately unchanged apart from sampling variation. Fixed-reference-PCA MMD² detects some distributional change but does not identify the affected gene relationships. In contrast, CellFidelity’s within-program coordination similarity decreases and its coordination error increases with shuffling strength (Figure 8B–C). The shared-shuffle control applies a single common cell-row permutation to the union of genes in all selected programs, largely retaining their joint coordination and producing smaller changes in these readouts. In the illustrated B/plasma comparison, clean-versus-corrupted classifier AUROC remains close to chance (~0.5), providing limited information about the source of the distortion.
Response to R3W1: Limited downstream validation
We thank the reviewer for highlighting the importance of establishing the downstream biological relevance of the identified fidelity defects. We agree that detecting discrepancies between generated and real cells does not necessarily demonstrate their consequences for biological inference. To address this concern, we conducted two complementary analyses: (1) controlled corruptions of real-cell expression profiles to characterize how specific fidelity defects affect downstream biological analyses, and (2) synthetic-cell augmentation experiments to evaluate whether different generative models can improve downstream inference.
1. Controlled corruptions reveal distinct, task-dependent consequences of fidelity defects.
We introduced four types of controlled corruptions into real-cell expression profiles, targeting different biological properties: canonical-marker detection, cell-type-specific functional programs, cell-state composition, and within-program gene coordination. These included canonical-marker dropout, nonmarker identity-program dropout, reference-subpopulation depletion, and independent shuffling of program-member expression across cells. By varying corruption strength and including matched controls where applicable, we examined how changes in CellFidelity measurements correspond to changes in downstream analyses, including differential expression (DE), pathway enrichment, and cell-state-associated transcriptional signals (Figure 6A–F).
The controlled perturbations revealed distinct, task-dependent consequences:
a) Marker fidelity (Figure 6A). Progressive canonical-marker dropout increased both the mean marker-detection error and the mean absolute error in marker-gene DE log2 fold changes. These parallel responses link impaired marker detection to errors in the corresponding downstream DE estimates.
b) Functional-program fidelity (Figure 6B–C). Nonmarker identity-program dropout altered the overall identity-program score contrast but caused comparatively modest errors in broad Hallmark enrichment (Figure 6B). Larger enrichment-score changes occurred in directly targeted Gene Ontology (GO) biological-process terms (Figure 6C). Because these GO terms contain manipulated genes, this comparison tests targeted analytical sensitivity rather than independently validating biological function. The contrast demonstrates that downstream sensitivity depends on the functional summaries being examined.
c) Semantic and population fidelity (Figure 6D–E). Progressive depletion of a plasma-cell state reduced state-associated transcriptional signals and population-level identity-program contrasts, whereas global manifold coverage remained relatively stable under moderate depletion. This highlights an important limitation of global distribution-based assessments: a dataset may retain reasonable overall coverage while losing biologically meaningful local cell states.
d) Gene coordination (Figure 6F). Independently shuffling expression values among genes within a program disrupted gene coordination similarity while leaving the targeted biological-process response largely unchanged. This illustrates that preservation of gene-level expression and pathway-level summaries does not necessarily imply preservation of the underlying gene–gene dependency structure.
Together, these results demonstrate that different fidelity defects affect distinct aspects of downstream biological analysis. Importantly, they also show that a dataset can appear reliable under one analytical criterion while exhibiting substantial defects under another, supporting the need for multidimensional fidelity assessment.