acceptodds
Under review as a conference paper at ICLR 2027

What Makes a Generated Cell Realistic? Dissecting Biological Fidelity in Single-Cell Generative Models

Abstract

Single-cell generative models are increasingly used to synthesize cellular profiles and predict unobserved biological states, but defining what makes a generated cell realistic remains an open problem. Existing evaluations often collapse realism into global statistical similarity, low-dimensional overlap, or classifier agreement, even though a generated cell may appear realistic in one sense while being biologically wrong in another. We propose CellFidelity, a multidimensional diagnostic framework that decomposes biological realism into four complementary dimensions: lexical fidelity of gene usage and expression magnitude, syntactic fidelity of gene relationships and functional programs, semantic fidelity of conditional cellular identity and identity-associated programs, and population fidelity of diversity and organization. CellFidelity operationalizes these dimensions using reference-calibrated measurements that distinguish deviations from the empirical variation observed among real cells. Across representative single-cell generators, CellFidelity exposes failure modes hidden by aggregate metrics: strong marker enrichment despite incomplete identity programs, plausible pathway activity despite sparse constituent-gene recovery, and globally similar populations despite distorted heterogeneity. These results show that generated-cell realism cannot be captured by a single notion of similarity. CellFidelity provides a diagnostic framework for revealing model-specific biological failures, guiding model development, and supporting more principled use of generated cells in downstream biology.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
single-cell RNA sequencing, generative models, generative model evaluation, biological fidelity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uRhdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work addresses evaluation of single-cell generative models. Compared to previous works, which focus on specific aspects of evaluation such as global statistical similarity, classifier agreement, or metrics derived from specific biological priors, this work proposes a more comprehensive, multidimensional framework for assessing biological fidelity. The framework covers different flavors of fidelity (lexical, syntactic, semantic, population) covering gene expression, functional programs, cellular identity, and population heterogeneity. Evaluation across several generative models reveals distinct failure modes that may be overlooked by more aggregate metrics.

Critical strengths
  • The problem setting is very relevant, as evaluating the biological fidelity of single-cell generative models is an increasingly important challenge. Several prior works address this setting, but existing metrics often capture different (and potentially complementary) aspects of fidelity. From this point of view, a comprehensive and integrative study can provide meaningful and novel insights.
  • The framework is well-structured, decomposing evaluation into four dimensions providing a useful organization of complementary biological properties. In particular, the use of reference-calibrated metrics is sensible, recognizing that unusually high values (e.g., marker enrichment or expression magnitude) can themselves indicate unrealistic behavior.
  • Comprehensive experimental characterization. The work spans different categories of models, biological metrics, and calibration controls.
Critical weaknesses
  • Limited downstream validation. The study identifies different discrepancies between generated and real cells across tested models, which could already support further metric-supported model development. However, it is not clear how and whether these discrepancies translate into meaningful differences into downstream biological applications, such as differential expression or pathway inference. Such analysis would further strengthen the proposed framework.
  • Limited comparison with existing approaches. Given that many metrics are based or inspired by existing techniques, with the main contribution being their organization into a unified framework, the authors should provide direct and explicit comparisons with existing approaches and fully clarify the additional insights derived using CellFidelity (i.e., their integration).
Questions for authors

See Weaknesses

6 comments

Response to R3W1: Limited downstream validation

We thank the reviewer for highlighting the importance of establishing the downstream biological relevance of the identified fidelity defects. We agree that detecting discrepancies between generated and real cells does not necessarily demonstrate their consequences for biological inference. To address this concern, we conducted two complementary analyses: (1) controlled corruptions of real-cell expression profiles to characterize how specific fidelity defects affect downstream biological analyses, and (2) synthetic-cell augmentation experiments to evaluate whether different generative models can improve downstream inference.

1. Controlled corruptions reveal distinct, task-dependent consequences of fidelity defects.

We introduced four types of controlled corruptions into real-cell expression profiles, targeting different biological properties: canonical-marker detection, cell-type-specific functional programs, cell-state composition, and within-program gene coordination. These included canonical-marker dropout, nonmarker identity-program dropout, reference-subpopulation depletion, and independent shuffling of program-member expression across cells. By varying corruption strength and including matched controls where applicable, we examined how changes in CellFidelity measurements correspond to changes in downstream analyses, including differential expression (DE), pathway enrichment, and cell-state-associated transcriptional signals (Figure 6A–F).

The controlled perturbations revealed distinct, task-dependent consequences:

a) Marker fidelity (Figure 6A). Progressive canonical-marker dropout increased both the mean marker-detection error and the mean absolute error in marker-gene DE log2 fold changes. These parallel responses link impaired marker detection to errors in the corresponding downstream DE estimates.

b) Functional-program fidelity (Figure 6B–C). Nonmarker identity-program dropout altered the overall identity-program score contrast but caused comparatively modest errors in broad Hallmark enrichment (Figure 6B). Larger enrichment-score changes occurred in directly targeted Gene Ontology (GO) biological-process terms (Figure 6C). Because these GO terms contain manipulated genes, this comparison tests targeted analytical sensitivity rather than independently validating biological function. The contrast demonstrates that downstream sensitivity depends on the functional summaries being examined.

c) Semantic and population fidelity (Figure 6D–E). Progressive depletion of a plasma-cell state reduced state-associated transcriptional signals and population-level identity-program contrasts, whereas global manifold coverage remained relatively stable under moderate depletion. This highlights an important limitation of global distribution-based assessments: a dataset may retain reasonable overall coverage while losing biologically meaningful local cell states.

d) Gene coordination (Figure 6F). Independently shuffling expression values among genes within a program disrupted gene coordination similarity while leaving the targeted biological-process response largely unchanged. This illustrates that preservation of gene-level expression and pathway-level summaries does not necessarily imply preservation of the underlying gene–gene dependency structure.

Together, these results demonstrate that different fidelity defects affect distinct aspects of downstream biological analysis. Importantly, they also show that a dataset can appear reliable under one analytical criterion while exhibiting substantial defects under another, supporting the need for multidimensional fidelity assessment.

AuthorsOpenReview ↗

Response to R3W1: Limited downstream validation (Continued)

2. Synthetic-cell augmentation reveals model-dependent downstream utility.

To further examine whether the fidelity differences identified by CellFidelity translate into practical differences in downstream applications, we evaluated synthetic-cell augmentation for DE and pathway inference. Using 800 real cells as a fixed reference, we compared three settings: 400 real cells alone (R400), the same 400 real cells augmented with 400 generated cells (R400+G400), and 800 generated cells alone (G800). The evaluation covered three cell-type contrasts and 10 paired technical resamples (Figure 7A–D).

We observed substantial differences across generative models. Adding scDiffusion-generated cells consistently reduced DE log2 fold-change reconstruction error relative to R400 alone, whereas CFGen provided smaller improvements (Figure 7A–B). In contrast, augmentation with scVI- and C2S-generated cells increased this error. These differences are broadly consistent with our main CellFidelity evaluation, in which scDiffusion showed stronger preservation of expression and functional-program properties, while the other models exhibited different fidelity deficiencies.

Importantly, improvements in DE reconstruction did not consistently lead to improvements in pathway inference (Figure 7C–D). This further emphasizes that downstream utility is task-dependent: synthetic cells that help reconstruct gene-wise expression differences may not necessarily preserve the coordinated biological signals required for reliable pathway-level interpretation.

These findings complement the controlled corruption experiments by demonstrating that fidelity differences are relevant not only under artificially introduced defects but also when evaluating the practical use of cells generated by different models.

Together, these findings support the importance of multidimensional fidelity assessment. Controlled corruptions connect specific fidelity defects to interpretable analytical consequences, while generated-cell augmentation shows that practical utility depends on both the model and the downstream task. We therefore interpret these experiments as evidence of downstream analytical relevance and model-dependent utility, rather than as universal predictive relationships between individual metrics and downstream performance.

Response to R3Q2: Limited comparison with existing approaches

We thank the reviewer for this important suggestion. We agree that CellFidelity should be evaluated in direct comparison with existing approaches, particularly because several of its constituent metrics are adapted from or inspired by established techniques. The main contribution of CellFidelity is not to introduce entirely new metrics, but to organize complementary measurements into a biologically interpretable framework that can characterize different failure modes of generated single-cell data. To clarify this contribution, we conducted systematic controlled-corruption experiments comparing CellFidelity with conventional evaluation measures and examining the additional diagnostic insights provided by their integration.

1. Direct comparison with established readouts.

We constructed nine controlled-corruption families using real B/plasma, endothelial, and neuronal cells, covering four aspects of biological fidelity: gene inclusion and expression magnitude (top-K transcriptome truncation, frequent-to-rare gene replacement, and detection-rate-stratified expression scaling), inter-gene dependence (within-program coordination shuffling), correspondence between expression and cellular identity (cell-type label mismatch, canonical-marker dropout, and identity-program dropout), and population heterogeneity (reference-subpopulation depletion and gene-wise variance compression).

Each corruption was applied at predefined strengths, with matched controls where applicable. The original, unperturbed real cells served as the clean baseline; matched controls were additional corruption-specific comparisons. Rather than assuming that each intervention should affect only one dimension, we examined both its intended effects and broader consequences across all 27 originally submitted CellFidelity metrics.

We compared these responses with four established evaluation readouts: mean gene-wise Wasserstein-1 distance (W1), mean absolute error in gene-wise zero fractions (ZeroMAE), squared maximum mean discrepancy (MMD²) in a fixed reference-PCA space, and held-out logistic-regression AUROC for distinguishing clean from corrupted cells. These measures represent conventional gene-level and distributional evaluation approaches rather than complete reproductions of published benchmarking frameworks. MMD² is also included among the original CellFidelity metrics and is reused here as a stand-alone global baseline.

This comparison was designed to distinguish two questions: whether an evaluation measure detects a discrepancy, and whether the combined measurements identify the biological property affected by that discrepancy. The aligned results are summarized in Figure 8A, with representative diagnostic contrasts in Figure 8B–D.

2. Examples where CellFidelity provides diagnostic information beyond individual discrepancy measures.

Gene coordination can deteriorate despite preserved gene-wise marginals. Independent within-program shuffling permutes each selected gene across cells, preserving its marginal expression distribution while disrupting cross-gene coordination. Accordingly, gene-wise W1 and ZeroMAE remain approximately unchanged apart from sampling variation. Fixed-reference-PCA MMD² detects some distributional change but does not identify the affected gene relationships. In contrast, CellFidelity’s within-program coordination similarity decreases and its coordination error increases with shuffling strength (Figure 8B–C). The shared-shuffle control applies a single common cell-row permutation to the union of genes in all selected programs, largely retaining their joint coordination and producing smaller changes in these readouts. In the illustrated B/plasma comparison, clean-versus-corrupted classifier AUROC remains close to chance (~0.5), providing limited information about the source of the distortion.

Reviewer_UJpROpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

AI-generated cell data may contain distorted biological relationships. This paper introduces CellFidelity. It evaluates generated cells along four dimensions: gene expression, functional programs, cell identity, and population heterogeneity. It calibrates these measurements against empirical distributions from real cells. By evaluating existing biological models, the authors identify several problems. These include capturing marker genes while missing broader cell characteristics, recovering only a few genes within a program, and lacking diversity in generated populations.

Critical strengths
  1. The biological reliability of generated single-cell data is an important problem. The authors argue that evaluating key biological structures is more meaningful than looking only at overall distributional similarity.
  2. The paper is clear and easy to read. The figures are well presented.
Critical weaknesses
  1. Why these four complementary dimensions? What are their boundaries, and how are they related? Could problems across several dimensions simply reflect the same underlying issue? What additional information does this organization provide beyond existing metric collections? For a paper focused on evaluation and analysis, I think the most important task is to carefully explain why these four dimensions form a meaningful analytical structure, rather than just a new grouping of existing metrics.
  2. The analysis mainly describes distortions and offers some tentative interpretations and suggestions. I would like to see a more detailed analysis of how these errors arise, along with practical guidance for improving existing models.
Questions for authors
  1. The y-axis label and numerical range in Fig. 4A seem to correspond to the raw marker fraction. However, the caption and Eq. 21 define enrichment normalized by the background frequency.
7 comments

Response to R2W1: The rationale, boundaries, and integration of four dimensions

We thank the reviewer for raising this important question. CellFidelity organizes biological fidelity around four complementary questions rather than statistically independent metric categories. We agree that a single underlying defect can affect multiple dimensions. The value of this organization lies in distinguishing different biological properties through their joint metric responses, as supported by our additional controlled-corruption experiments and direct comparisons with established evaluation measures.

1. Why these four complementary dimensions? What are their boundaries, and how are they related?

The natural-language analogy in our Introduction provides a conceptual motivation for this organization. Specifically, the four dimensions address distinct biological questions:

  • Lexical: Are the appropriate genes detected, and are their expression magnitudes plausible?

  • Syntactic: Are genes assembled into supported, complete, and coordinated functional programs?

  • Semantic: Does the expression profile correspond to the requested cell identity, including canonical markers and broader identity-associated functions?

  • Population: Does the collection of cells preserve the distribution, coverage, and heterogeneity of real cellular states?

Their boundaries are defined by the biological questions being evaluated, not by mutually exclusive gene sets or statistics. For example, syntactic fidelity concerns the internal reconstruction of functional programs, whereas semantic fidelity evaluates whether these programs are appropriate for the specified cellular identity. A program may therefore be internally coherent but inconsistent with its assigned cell type. Likewise, individually plausible cells do not necessarily reproduce population-level diversity.

2. Could problems across several dimensions reflect the same underlying issue?

To examine the relationships among these dimensions, we introduced nine controlled-corruption families into real B/plasma, endothelial, and neuronal cells. These included transcriptome truncation, frequent-to-rare gene replacement, and expression scaling to alter gene representation or magnitude; within-program gene shuffling to disrupt gene coordination; cell-type label mismatch and targeted marker or identity-program dropout to alter conditional identity; and reference-subpopulation depletion and gene-wise variance compression to modify population structure.

Each intervention was applied at predefined strengths, with matched controls where appropriate. We evaluated its responses across all 27 originally submitted CellFidelity metrics, examining both the intended effects and changes in other dimensions.

The results demonstrate that the dimensions are complementary but not independent (Figure 2). For example, transcriptome truncation reduces both gene-level coverage and functional-program breadth, whereas cell-type label mismatch affects identity measurements and can alter population statistics evaluated within assigned cell types. Conversely, targeted disruptions can affect one property while leaving related measurements comparatively preserved: independent program-gene shuffling reduces coordination without substantially changing gene-level expression percentiles, and identity-program dropout weakens broader identity contrast while largely preserving canonical-marker contrast (Figure 4).

These findings show that a single disturbance may propagate across dimensions, but deterioration in one property does not necessarily imply equivalent deterioration in another. The metric co-response matrix (Figure 3) further describes these relationships without treating correlation as evidence of statistical independence or causal equivalence.

AuthorsOpenReview ↗

Response to R2W1: The rationale, boundaries, and integration of four dimensions (Continued)

3. What does the organization add beyond existing metric collections?

We also compared the same nine controlled-corruption conditions with four representative conventional readouts: mean gene-wise Wasserstein-1 distance, gene-wise zero-fraction error, MMD² in a fixed reference-PCA space, and held-out clean-versus-corrupted classification AUROC (Figure 8A–D). These are representative discrepancy measures rather than reproductions of complete published evaluation frameworks; MMD² is also one of the submitted CellFidelity population metrics.

The conventional measures detected several perturbations, particularly those affecting gene representation or global expression distributions. However, the joint CellFidelity profile provided more specific interpretations of certain defects. For instance, independent shuffling of program-member genes preserves gene-wise marginal distributions, leaving Wasserstein-1 distance and zero-fraction error approximately unchanged, while program coordination similarity decreases and coordination error increases. Although PCA-space MMD² can detect a distributional change, it does not by itself identify the disrupted gene relationships (Figure 8B–C).

Similarly, selective subpopulation depletion predominantly reduces manifold coverage while relatively preserving precision, indicating missing reference-supported states. In contrast, cell-type label mismatch disrupts expression–identity correspondence without changing the pooled expression matrix (Figure 8D). These comparisons highlight the value of organizing complementary measurements around biological questions: aggregate metrics can indicate that generated data deviate from the reference, while the CellFidelity profile helps localize the discrepancy to specific properties, such as gene coordination, conditional identity, or population-state coverage.

4. Summary

Together, these analyses support the four-dimensional organization as a biologically interpretable structure for evaluating generated cells. By explicitly connecting gene-level plausibility, functional-program organization, cellular identity, and population heterogeneity, CellFidelity enables the joint interpretation of affected and relatively preserved properties and provides a systematic way to characterize distinct biological failure patterns.

We do not assume that the dimensions are statistically independent or that their combined profiles uniquely identify an underlying causal mechanism. We will clarify these boundaries in the Introduction and Discussion and include the controlled-corruption comparisons and implementation details in the revised Supplementary Materials.

Response to R2W2: The origins of discrepancies and practical guidance for model improvement

We thank the reviewer for emphasizing the need to connect observed fidelity defects with their possible origins and practical implications for model improvement. We agree that identifying discrepancies alone is insufficient. To address this concern, we conducted two complementary analyses: controlled downstream validation to examine how specific fidelity defects affect biological analyses, and model-specific investigations to identify plausible sources of observed discrepancies and guide targeted improvements.

1. Controlled perturbations reveal task-dependent downstream consequences.

To establish the biological relevance of fidelity discrepancies, we introduced four controlled perturbations into real-cell expression profiles: canonical-marker dropout, nonmarker identity-program dropout, reference-subpopulation depletion, and within-program gene coordination shuffling. These interventions target different aspects of cellular identity, functional programs, cell-state composition, and gene dependence. Each was applied at increasing strengths, with matched controls where applicable. We evaluated the corresponding CellFidelity measurements alongside downstream differential-expression, pathway-inference, and cell-state-associated readouts (Figure 6A–F).

The results revealed distinct analytical consequences. Marker dropout increased differential-expression effect-size errors, while identity-program dropout produced clearer changes in directly affected Gene Ontology biological processes than in broader Hallmark pathways. Progressive plasma-state depletion altered state-associated transcriptional signals and identity-program contrast even when global manifold coverage remained comparatively stable. In contrast, gene coordination shuffling substantially disrupted program coordination while leaving the targeted pathway-enrichment response largely unchanged. This illustrates that conventional pathway-enrichment analysis may remain insensitive to disruptions in the internal relationships among pathway genes.

These results demonstrate that different fidelity defects can have different downstream consequences, and that preserved performance in one analytical task does not necessarily imply preservation of other biological properties.

2. C2S highlights the need to distinguish output limitations from expression reconstruction.

C2S provides an example of how CellFidelity can separate multiple contributors to an observed deficiency. Expanding the output from approximately 200 to 1,000 genes improved gene, marker, and functional-program coverage, indicating that restricted output support contributes substantially to these deficits. However, higher-order discrepancies in program coordination and population heterogeneity were not fully resolved by increased gene coverage.

We further performed matched-processing experiments using real cells subjected to equivalent top-K truncation, comparing expression magnitudes before and after library-size normalization (Figure 5). The originally elevated C2S expression percentiles were substantially reproduced by applying the same truncation and normalization to real cells. Before normalization, however, C2S exhibited lower reconstructed expression magnitudes than matched truncated real cells, suggesting additional contributions from rank-to-expression reconstruction, generated gene selection or ranking, or their combined effects.

These findings motivate distinct improvement strategies: expanding output coverage, independently evaluating expression reconstruction and normalization, and examining higher-order biological structure after controlling for gene support. The downstream augmentation analyses further showed that the evaluated C2S variants increased differential-expression reconstruction errors, emphasizing that improvements in individual fidelity measurements should be validated against actual downstream performance.

Reviewer_dXy4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles a great problem as the field is aiming to build virtual cells and is arguably measuring them completely wrong. The proposed cellfidelity evluation framework breaks realism into 4 dimensions using an analogy to natural language. The key design principle is reference calibration. Metrics are read against an internal real validation subset and an independent real control, so a deviation in either direction counts as miscalibration. For example, marker enrichment above the real range is treated as a failure. The framework was applied to several models with thorough analysis on each.

Critical strengths
  1. Real-data calibration provides a useful basis for interpreting scores. Comparing generated outputs with two real-cell reference sets helps distinguish plausible values from departures that might otherwise appear favourable. In particular, elevated marker enrichment should be assessed against real-cell behaviour rather than interpreted as an improvement by default. This is a persuasive design principle, although its value depends on how the reference sets are constructed and how departures are defined.

  2. The framework offers detailed, inspectable descriptions of generated outputs. Reporting gene support, program structure, identity contrasts, and population properties separately reveals discrepancies that a single summary score could obscure. For example, the reported scVI results distinguish agreement in gene counts from agreement in gene selection, while the C2S comparison distinguishes improved coverage from recovery of coordination and dimensionality. These observations demonstrate descriptive value, even though their biological specificity and attribution require further testing. The appendix’s detail on thresholds, seeds, subsampling, and tie handling also provides a useful foundation for reproducibility and potential community adoption.

Critical weaknesses
  1. The intended interpretation of the metrics is insufficiently validated. The paper assigns biological meanings to the four dimensions and presents them as complementary, but it does not directly test how reliably the metrics respond to known changes in real cells. Controlled-corruption experiments would help establish this. Examples include:
  • Gene dropout or truncation to the top K genes.

  • Shuffling expression across cells for individual program members, preserving gene-wise marginal distributions while disrupting coordination.

  • Deliberate mismatches between cell-type labels and expression profiles.

  • Selective removal or underrepresentation of biological subpopulations, with evaluation sample size controlled.

These experiments should establish sensitivity to the intended perturbations and characterise effects on other metrics. The dimensions need not be statistically independent, but their relationships and added value should be demonstrated. An analysis of metric correlations across sufficiently varied conditions would also help identify redundancy.

  1. The generator comparisons provide examples of differing metric profiles, but they do not establish which underlying failures caused those differences. Links to downstream tasks such as differential expression, pathway inference, and augmentation are also left to future work, limiting conclusions about practical utility.

The generator comparison does not adequately separate model behaviour from processing and evaluation choices.

  • C2S magnitude findings depend on reconstructed expression values. Appendix B.5 describes a population-level rank-to-expression mapping, while Sections B.1 and 4.1 describe library-size normalisation. With only K genes retained, the reconstruction and rescaling steps can affect the apparent abundance of those genes. Consequently, the elevated expression and marker-enrichment results may reflect a combination of gene selection, expression reconstruction, and normalisation. The current analysis does not sufficiently distinguish these contributions. Real-cell controls subjected to comparable truncation and processing would help assess this.

  • The separation between training and evaluation data needs clarification. As I read Appendix B.1, the generators are trained on the full reference, including the internal real-validation subset. If so, favourable agreement with this subset does not establish fidelity on held-out data, and memorised training examples could receive favourable scores. The independent real control may provide additional evidence, but its relationship to training cells, datasets, donors, and batches needs to be explicit.

  • The conclusions need to remain specific to the evaluated implementations. scDiffusion uses a count-output adaptation, and C2S is retrained from a Pythia-160M base model rather than evaluated through the released C2S or C2S-Scale checkpoints. These choices may be reasonable for the experiment, but their effects and the comparability of training and tuning effort are not sufficiently characterised. The reported ranking therefore applies to these implementations and pipelines; its generality to the published methods remains uncertain.

  • The main analysis covers a limited subset of cell types. The focus on three of the 18 harmonised types leaves uncertainty about how consistently the conclusions hold across the reference.

These issues do not remove the descriptive value of the framework, but they weaken the attribution of observed discrepancies to particular generators.

Questions for authors
  1. Can you provide controlled-corruption experiments showing how the metrics respond to known perturbations of real cells? Please characterise both intended responses and effects on other dimensions. This is the most important issue for my assessment.

  2. What is the exact sequence of rank-to-expression mapping, back-transformation, truncation, and library-size normalisation for C2S? How sensitive are the magnitude findings in Figures 2E, 3C, and 4B to these choices? Do comparable effects appear when real cells undergo matched truncation and processing?

  3. Please clarify the overlap between generator training data, the internal real-validation subset, and the independent real control. In addition to cell-level overlap, what dataset, donor, or batch overlap exists? Have you assessed similarity or near-duplication between generated and training cells?

8 comments
AuthorsOpenReview ↗

Response to R1W1&Q1: Controlled-corruption validation

Response to R1W1&Q1: Controlled-corruption validation

We thank the reviewer for this important suggestion. We agree that comparisons among generative models alone do not establish how fidelity metrics respond to known biological distortions. To address this concern, we conducted controlled-corruption experiments using real B/plasma, endothelial, and neuronal cells, applying nine predefined corruption families and evaluating all 27 originally submitted CellFidelity metrics without altering their definitions. These experiments assess whether the metrics respond to their intended distortions and whether their joint responses can distinguish different biological failure modes.

1. Nine controlled-corruption families cover four aspects of biological fidelity.

a) Gene inclusion and expression magnitude (Lexical). Three interventions were used: top-K transcriptome truncation (K = 1,000, 500, or 200), frequent-to-rare gene replacement while preserving the number of detected genes, and detection-rate-stratified expression scaling (×0.5 or ×1.5) followed by library-size renormalization.

b) Inter-gene dependence (Syntactic). Within-program coordination shuffling independently permuted expression values across cells for selected program-member genes, preserving their gene-wise marginal distributions while disrupting gene coordination. The shared-shuffle control instead applied a common cell-row permutation to the union of genes in all selected programs, largely preserving their joint relationships.

c) Correspondence between expression and cellular identity (Semantic). Three interventions were used: cell-type label mismatch without altering the underlying expression matrix, canonical-marker dropout, and nonmarker identity-program dropout. These interventions distinguish incorrect identity assignments, marker loss, and broader identity-program disruption.

d) Population support and heterogeneity (Population). Reference-subpopulation depletion progressively removed a selected real-cell substate, while gene-wise variance compression reduced intercellular expression variability among highly detected genes.

Each intervention was applied at predefined strengths, with repeated random seeds where applicable. Unmodified real cells served as the clean baseline, and matched controls were included to account for potential confounding effects from processing, replacement, permutation, or changes in sample size.

2. The metrics respond to their intended biological distortions.

Across the three evaluated cell types, the targeted CellFidelity metrics generally changed in the expected directions as corruption strength increased (Figure 1A–I).

a) Transcriptome truncation reduced expected-gene recall and functional-program breadth. Frequent-to-rare replacement reduced recovery of reference-expected genes while increasing the representation of infrequent genes among the top 200 expressed genes, without reducing detected-gene counts. Detection-rate-specific expression scaling shifted positive-expression percentiles in the targeted bins (Figure 1A–C).

b) Independent within-program shuffling progressively reduced coordination similarity and increased coordination error, whereas the shared-shuffle control produced comparatively smaller changes (Figure 1D).

c) Cell-type label mismatch, canonical-marker dropout, and identity-program dropout impaired their corresponding identity measurements (Figure 1E–G).

d) Finally, subpopulation depletion reduced manifold coverage, while gene-wise variance compression produced the expected variance-bias responses in the targeted high-detection-rate gene strata (Figure 1H–I).

These results demonstrate that the submitted metrics are sensitive to the biological properties they were designed to evaluate, with intervention- and cell-type-dependent responses.

AuthorsOpenReview ↗

Response to R1W1&Q1: Controlled-corruption validation (Continued)

3. Controlled corruptions reveal both cross-dimensional effects and complementary metric responses.

We further examined how each corruption affected the complete set of 27 metrics rather than only its intended readouts. The 9 × 27 response heatmap (Figure 2) revealed that several corruptions affected multiple dimensions simultaneously. For example, transcriptome truncation reduced both gene-level coverage and functional-program reconstruction, while cell-type label mismatch affected identity measurements and could alter population statistics evaluated within assigned cell types.

These cross-dimensional responses are biologically plausible because the four dimensions describe complementary properties rather than statistically independent processes. We additionally examined correlations among metric responses across graded corruption conditions (Figure 3). This analysis provides a descriptive view of metric relationships without implying statistical independence, redundancy, or causal equivalence.

Importantly, targeted corruptions did not necessarily produce comparable changes in all related measurements. Representative paired comparisons (Figure 4A–H) illustrate four distinct patterns:

a) Frequent-to-rare replacement reduced expected-gene recall while increasing the fraction of infrequent genes among the top 200 expressed genes, despite preserving the total number of detected genes (Figure 4A–B). This demonstrates that the two metrics capture complementary aspects of gene composition and representation.

b) Independent program-gene shuffling disrupted coordination similarity while leaving positive-expression percentiles comparatively stable (Figure 4C–D).

c) Nonmarker identity-program dropout reduced broader identity-program contrast while largely preserving canonical-marker contrast (Figure 4E–F).

d) Selective subpopulation depletion reduced manifold coverage without a corresponding reduction in precision (Figure 4G–H).

These comparisons demonstrate that substantial deterioration in one biological property can coexist with relatively preserved measurements of another. Jointly examining both affected and preserved readouts therefore provides a more informative characterization of the underlying fidelity defect.

4. Comparisons with existing readouts further support the diagnostic value of integration.

We further compared the same nine controlled-corruption conditions with four conventional readouts: mean gene-wise Wasserstein-1 distance (W1), gene-wise zero-fraction error (ZeroMAE), reference-PCA MMD², and clean-versus-corrupted classification AUROC (Figure 8A–D). Although these measures detected several corruptions, the combined CellFidelity profile provided more specific information about certain failure modes.

For example, independent program-gene shuffling disrupted gene coordination while leaving gene-wise W1 and ZeroMAE largely unchanged (Figure 8B–C). Similarly, reference-subpopulation depletion and cell-type label mismatch could both produce changes in MMD² within the evaluated B/plasma group, but exhibited distinct metric profiles: depletion primarily reduced manifold coverage while preserving precision, whereas label mismatch altered expression–identity correspondence without changing the pooled expression matrix (Figure 8D). Thus, comparable global discrepancy scores can reflect different biological failures, which become distinguishable through complementary identity and population measurements.

These comparisons further support the diagnostic value of integrating complementary metrics, rather than relying on individual discrepancy scores alone.

Together, the controlled-corruption experiments provide evidence that CellFidelity can detect predefined biological distortions, characterize their cross-dimensional consequences, and distinguish affected from relatively preserved biological properties. We emphasize that these metric profiles support diagnostic interpretation rather than unique causal identification. We will include the complete corruption definitions, matched controls, dose-response results, and comparisons with conventional readouts in the revised Supplementary Materials.

AuthorsOpenReview ↗

Response to R1W2&Q2: C2S expression reconstruction and matched-processing controls

Response to R1W2&Q2: C2S expression reconstruction and matched-processing controls

We thank the reviewer for highlighting the potential effects of rank-to-expression reconstruction and subsequent library-size normalization on C2S expression magnitudes. We agree that these processing choices may substantially influence the observed expression magnitudes. We therefore conducted additional matched-processing experiments to examine their contributions.

Processing procedure. In our C2S implementation, raw counts were library-size normalized and log-transformed before each cell was converted into a sequence of genes ranked by expression. At generation, K = 200 or 1,000 specified the requested number of genes but did not guarantee K valid, distinct genes in the output. We removed invalid gene names and collapsed duplicate genes according to their average positions in the generated sequence.

We assigned expression values to retained genes using a population-level rank-to-expression fit function derived from preprocessed real cells’ expressions and ranks:

x~g=fref(rg)\tilde{x}_g = f_{\mathrm{ref}}(r_g)

Same as the original work, each retained gene was assigned the expression value corresponding to its generated rank, while genes absent from the generated sequence were assigned zero. All cell types share the same fit function. The reconstructed values were subsequently processed into a normalized expression matrix using a back-transformation, library-size normalization to 10,000, and log1p transformation for evaluation. We verified the consistency of these processing steps and will provide the exact implementation and updated evaluation results in the revised manuscript.

Matched-processing controls. We constructed five comparison groups: R0, full real-cell expression; R1, real cells truncated to their top-K expressed genes without further normalization; R2, the same truncated real cells after library-size normalization; C0, rank-mapped C2S expression before final normalization; and C1, the normalized C2S expression from the original evaluation pipeline. For C2S-200, we matched the actual output support of 198 genes in the real-cell controls.

We performed three comparisons:

(i) R2 vs. C1: Real and generated cells after matched gene truncation and normalization.

(ii) R1 vs. C0: Expression magnitudes before final normalization.

(iii) R0 vs. C0: Reconstructed C2S expression relative to full real-cell profiles as a contextual reference.

Results. The matched-processing comparisons revealed three main findings (Figure 5A–D).

First, the elevated expression percentiles originally observed for C2S were substantially reproduced by applying equivalent truncation and normalization to real cells. After normalization, both C2S and matched truncated real cells exhibited expression percentiles near the upper limit of the reference distribution, with broadly comparable positive-expression magnitudes (Figure 5A–B). These findings suggest that normalization under restricted gene support can contribute substantially to elevated expression percentiles.

Second, the original pre-normalization comparison showed lower expression percentiles for C2S than for matched truncated real cells (Figure 5A). This observation provides a basis for further examining the potential contributions of rank-to-expression reconstruction and generated gene selection or ranking.

Third, deficiencies in marker detection and functional-program member coverage remained evident relative to full real-cell profiles. The matched truncation controls showed that restricted output support could reproduce much of this under-coverage, while expression magnitudes among retained genes followed distinct patterns before and after final normalization (Figure 5C–D).

Together, these analyses indicate that restricted output support and normalization need to be considered when interpreting the C2S fidelity profile. We have verified the expression-processing pipeline and repeated the affected analyses using a consistent implementation. We will update the corresponding Results and Discussion, clarify the reconstruction procedure, and include the matched-processing controls in the revised Supplementary Materials.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.