acceptodds
Under review as a conference paper at ICLR 2027

Attribution-based diagnostics for single-cell foundation models

Abstract

Single-cell RNA foundation models (scFMs) are increasingly used for _in silico_ biological discovery, yet predictive performance alone does not reveal whether their decisions rely on biologically meaningful signals or model-specific shortcuts. We present a systematic post-hoc attribution study of downstream predictions in two applications: cell-type annotation with pretrained Cell2Sentence and scGPT models, and a controlled scGPT+STATE perturbation-prediction case study. For cell annotation, our analyses connect predictions to established marker genes while examining how gene relevance varies with model architecture, expression rank, and the surrounding expression profile. In a synthetic perturbation prediction setting, we compare attribution patterns with known regulatory ground truth to assess what they reveal about the model's learned computations. Across these tasks, attribution exposes biological feature use, higher-order dependencies, pathway-specific computation, learning progression, prediction reliability, and potential failure modes that are not fully captured by aggregate output metrics. Experiments with complete and sparse regulatory knowledge further show that the conclusions drawn from attribution depend critically on the chosen biological reference, baseline, and null model. Together, these results establish attribution as a flexible post-hoc diagnostic for understanding, debugging, and guiding the development of scFMs.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
transcriptomics, explainability, attribution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QpQ3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper examines whether single-cell foundation models use biologically meaningful signals for downstream predictions. This is an important topic. Using LRP, the authors analyze marker-gene relevance in C2S and scGPT classifiers and apply symbolic explanations to investigate contextual contributions of shared markers. Furthermore, a synthetic scGPT+STATE perturbation study compares attribution signatures with known regulatory relationships, differential expression, and input-ablation baselines. The findings support attribution as a diagnostic tool, while showing that apparent biological alignment depends strongly on reference selection and null models, particularly controls preserving regulatory-network degree.

Critical strengths
  1. The paper connects scFM downstream predictions to input genes across multiple pretrained models and two tissues. Comparisons with alternative attribution methods and input ablations provide useful support for the selected explanations.
  2. The degree-preserving GRN controls reveal that successful sparse-prior recovery need not identify genuine regulatory relationships. This is a valuable methodological trial for evaluating biological interpretability.
Critical weaknesses
  1. The biological meaning of the attribution quantities is insufficiently established. Control-expression relevance, intervention effects in a latent system, and predictions of normalized expression represent different quantities. Similarly, filtering symbolic relevance terms does not by itself establish what the model would predict after removing contextual genes.
  2. The incremental diagnostic value and scope remain limited. Perturbation LRP largely tracks predicted DE and perturbation ablation (Fig. H.1). The perturbation study also uses synthetic pretraining with both contexts and all perturbation identities represented during training, leaving the contributions of pretrained representations and supervised task learning insufficiently separated.
Questions for authors
  1. The cell-type analysis reports marker recovery, expression-rank effects, and contextual relevance across Cell2Sentence and scGPT. The connection between these observations and the biological interpretation of model behavior remains unclear. What is the principal biological conclusion established by this analysis beyond recovering known cell-type markers?
  2. Given the focus on Cell2Sentence and scGPT, what evidence supports extending these conclusions to other single-cell foundation models, such as Geneformer, and which claims should remain restricted to the evaluated configurations?
  3. The synthetic regulatory effects are defined in a latent dynamical system, whereas the predicted expression values reflect nonlinear count generation and library-size normalization. These transformations can change response magnitudes and rankings across genes. What evidence supports using the latent regulatory effects as the reference for evaluating attributions of normalized expression?
  4. The paper primarily demonstrates the framework’s effectiveness through aggregate statistical results, leaving its practical value for biological interpretation insufficiently illustrated. The authors should provide concrete biological cases demonstrating how the attribution results support a biologically meaningful interpretation of a model prediction.
1 comment

W1: Biological meaning of attribution quantities

We agree that these quantities capture distinct aspects of the system. Control-expression relevance measures contribution to predicted expression, and should not be treated as an intervention effect. Perturbation relevance attributes the prediction to the explicit perturbation input. We compare these attribution signatures with latent regulatory references as alignment diagnostics, not as equivalent biological quantities. The relationship between latent effects and observed expression is addressed in point Q3 below. We will clarify the interpretation of each quantity in the manuscript.

W1: Symbolic XAI

We agree that filtering relevance terms is not equivalent to explicitly removing genes from the input. The value of our Symbolic XAI analysis lies in distinguishing relevance confined to shared markers from relevance arising through their interactions with other genes, without modifying the original input and potentially introducing distribution shifts. This allows us to investigate how the model exploits shared markers in different cellular contexts rather than predict what would happen if other genes were removed. We have clarified this distinction in the revised manuscript.

W2: Diagnostic value and scope

While perturbation LRP shows agreement with predicted DE and perturbation ablation, our learning-progression analysis (Fig. 6A) demonstrates its complementary diagnostic value. Specifically, at epoch 5, perturbation LRP already shows regulatory alignment beyond random-model controls, whereas predicted DE remains indistinguishable from these controls. This suggests that attribution can provide an early alignment diagnostic to supplement output-level statistics.

We also evaluate both a fully random model and a pretrained scGPT backbone with an untrained STATE head. Since scGPT remains frozen during the first five fine-tuning epochs, the epoch-5 results support the emergence of regulatory alignment (beyond random-model controls) through supervised training of STATE on fixed pretrained representations. While these experiments do not fully disentangle the contributions of pretraining and fine-tuning, they help characterise how regulatory information emerges during training.

The added OOD analysis further shows approximately stable training-GRN alignment in perturbation-LRP despite declining predicted-DE alignment with that same GRN, illustrating complementary views of model generalisation.

Q1: Biological interpretation of model behaviour

Beyond marker recovery, our findings suggest that models distinguish closely-related cell subtypes by using shared marker genes in combination with the surrounding cellular context. Expression-rank dependence and weaker marker alignment in incorrect predictions further qualify the biological interpretation of these attributions. These analyses reveal how models exploit the biological information available in their input features and highlight potential failure modes not apparent from predictive performance alone. Thus, our principal contribution is to demonstrate how attribution-based diagnostics can assess the biological plausibility of model behaviour and inform model development, rather than uncover new biological relationships.

Q2: Generalisation of the findings to other scFMs

As with other interpretability studies, our goal is to characterise the behaviour of the specific models under investigation, rather than establish universal properties. While our study covers two distinct scFM architectures, with multiple pretrained backbones, model sizes, and context configurations, we do not claim that the observed attribution patterns generalise to all scFMs. The proposed analysis protocols can be applied to other architectures such as Geneformer, provided that suitable attribution methods are implemented and validated. Accordingly, our empirical findings regarding marker recovery, expression-rank effects, and contextual relevance remain specific to the evaluated configurations.

Q3: Use of latent reference

Our empirical-DE baseline signature is computed from normalised expression. Figures 5B–C shows precisely its strong agreement with the latent direct-response signs and stationary-effect rankings, empirically supporting these references after count generation and normalisation. This does not imply preservation of effect magnitudes or exact rankings.

Q4: Concrete biological cases

For example, in the CM CD4 versus EM CD4 comparison for C2S (Fig. 4), relevance confined to the shared-marker set (CD4 and IL7R) is near zero, whereas full-context relevance shifts toward EM CD4. IL7R is also more abundant and ranks earlier in EM CD4 inputs. Given the model’s tendency to assign greater relevance to earlier-ranked genes (Fig. 3), this example illustrates how attribution can reveal context-dependent feature use while identifying expression rank as a potential confounding factor.

Reviewer_UAxaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the authors present a systematic post-hoc attribution study of two applications: cell-type annotation with pretrained Cell2Sentence and scGPT models, and a controlled scGPT+STATE perturbation-prediction case study by using LRP and high-order LRP.

The proposed framework serves as a practical tool for analyzing the biological explainability of the scFMs instead of just focusing on the metric scores. In both applications, gene-level and context-level attribution analyses are performed to explain the complex relevance of input through the model, which reflects the complexity of biological interactions.

Critical strengths

The strength is about the gene-level and context-level attribution analysis, which shows the authors’ awareness of the complexity of biological signals. Besides, introducing XAI to the scFMs is a quiet good vision while more and more large models are developed for different biological tasks.

Critical weaknesses

Though the authors realize the complexity of biological signals, the experimental settings are quite simple comparing to the real data. For example, the simulated GRN data used in perturbation experiment can’t generally reflect the interplay and hierarchical regulatory relationships among epigenetic regulation, transcription factors (TFs), and complex intracellular signaling pathways. As such, the mapping between perturbation results and intrinsic GRN is over-simplistic and does not necessarily reflect the complex biochemical processes. Consequently, I have doubts about the effectiveness of this framework as a measurement for real biological discoveries when applied to real data.

Questions for authors

Have you considered using more complex data to conduct the attribution analysis? Could the authors improve the clarity of the manuscript?

2 comments

Simulator realism

We agree that real biological systems involve substantially more complex regulatory mechanisms than those represented in our synthetic setting. However, our choice of synthetic data is deliberate and motivated by the lack of reliable ground-truth regulatory relationships in real perturbation datasets. The explicit regulatory structure of the simulator, adopted from Qi and Chapfuwa (2026) and Lorch et al. (2026), enables controlled evaluation of diagnostic hypotheses against appropriate null controls, which would otherwise be difficult to perform on real data.

Our objective is to demonstrate how attribution-based diagnostics can be used to analyse whether models learn pre-specified, biologically meaningful relationships, how they use these relationships to make predictions, and where their behaviour deviates from biological expectations. Even when derived from controlled synthetic settings, such insights can be valuable for model development, as they can uncover limitations not apparent from predictive performance alone.

To extend our investigation, we have added experiments evaluating the model under systematically increasing regulatory distribution shifts within the same simulator, ranging from partial modifications of the training GRN to entirely independent GRNs (Section 4.4 and Appendix H.3). These experiments demonstrate how attribution-based diagnostics can reveal persistent associations with the regulatory structure learned during training, even when these associations become less apparent in model predictions, thereby providing complementary insights into model generalisation and its limitations.

While these experiments do not establish transferability to real perturbation data, they demonstrate the potential of regulatory-alignment diagnostics for identifying learned model behaviour and informing model development. Extending these analyses to real perturbation data using complementary regulatory evidence is an important direction for future work. Moreover, our cell-type analyses already demonstrate the diagnostic value of attribution on real biological data from blood and brain tissues.

Manuscript clarity

We have expanded the limitations paragraph to make the scope of the synthetic experiments explicit. We have also clarified the interpretation of contextual attribution to distinguish it from gene-removal predictions. We welcome specific pointers to other parts of the manuscript that would benefit from further clarification.

I appreciate the proposed framework, which provides a concrete and systematic approach to analyzing model interpretability. I believe the framework itself is valuable and has the potential to facilitate a better understanding of model behavior. However, I still have concerns about its applicability and practical utility on real-world data. Therefore, I have decided to maintain my original score.

Reviewer_SUfYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper assesses whether attribution-based explainability on scFMs are consistent with established biological knowledge and whether there are any disconnects and shortcut behavior of such attribution-based approaches. The paper uses scGPT and Cell2Sentence models for cell-type classification and apply LRP, in which the results show that LRP generally outperforms the expression-value baseline. For perturbation prediction, the paper also creates a synthetic dataset with a fully known regulatory structure and attempt to recover the regulatory relationships with LRP. They compared the results against differential-expression baselines and randomized controls.

Critical strengths

The experiments are rather comprehensive, examining several architectures and prediction tasks and experiment design choices. The paper also evaluates the faithfulness of LRP against alternative explainability methods.

The paper is written in a clear and organized manner.

Critical weaknesses

Synthetic dataset is a good direction, but is much weaker than real-world applications. To provide more value to the synthetic data experiments, I would suggest using multiple sets of synthetic data, and look at various data generation settings to provide more insights to the community.

Given that the paper primarily applies existing techniques rather than proposing a new method, I would like to see more experiments and discussion on the analysis of LRP. For example, the early-training analysis and the association between relevance concentration and prediction quality is interesting, and I would like to see more analyses on scenarios where attribution provides actionable information that cannot be obtained from the alternatives, or vice versa.

Questions for authors

Please address weaknesses above.

2 comments

W1: Extended synthetic evaluation under regulatory distribution shifts

We thank the reviewer for this suggestion. Our motivation for using synthetic data is that reliable ground-truth regulatory relationships are generally unavailable in real perturbation datasets, making it difficult to directly assess whether model attributions reflect the underlying regulatory mechanisms. To address the reviewer’s suggestion of exploring multiple synthetic datasets and generation settings, we have extended our experiments to systematically varied out-of-distribution (OOD) regulatory networks, with 80%, 50%, and 0% direct-edge overlap with the context-0 training GRN, as well as independently generated GRNs, using 10 random replicates per setting (Section 4.4 and Appendix H.3 in the updated manuscript, highlighted in colour).

We find that, as the regulatory structure changes, both predictive performance and attribution (perturbation- and gene-LRP) alignment with the new GRNs deteriorate. However, attribution alignment with the original training GRN remains relatively stable, whereas predicted DE shows substantially weaker alignment with that GRN under increasing distribution shift. This suggests that attribution can reveal persistent associations with the regulatory structure learned during training, even when these associations become less apparent in the model’s predicted responses. Thus, beyond evaluating model performance across different regulatory structures, these experiments demonstrate how attribution can provide complementary insights into model behavior under distribution shift.

W2: Attribution-based insights for model understanding and development

Our learning-progression analysis (Fig. 6A) shows that perturbation-LRP detects regulatory alignment at early training stages, before predicted DE becomes distinguishable from random-model controls. This highlights its potential for monitoring the emergence of regulatory alignment during training. We also find that perturbation-input relevance remains associated with prediction quality (Spearman ρ\rho = 0.666) after adjusting for predicted-DE standard deviation, predicted-expression zero fraction, and predicted-expression L2 norm (Section 4.4), suggesting its potential value for assessing prediction reliability (at least in-distribution).

Together with the new OOD experiments described above, these findings demonstrate how attribution complements output-based analyses by providing insights into learned regulatory associations, training progression, and prediction reliability. For example, persistent training-GRN attribution alignment cautions against interpreting poor OOD predictions as evidence that the model has adapted to the new regulatory structure.

I thank the authors for the added experiments in the revisions as well as the clarifications, some of my concerns have been addressed. It is a paper that primarily applies existing techniques rather than proposing a new method, but I think the results are worth sharing to the scFMs community. I am thus increasing my score and hoping that this translates into real-world data instead of simulated in future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.