acceptodds
Under review as a conference paper at ICLR 2027

A Mechanistic Probe of Regulatory Structure in Single-Cell Foundation Models

Abstract

Do single-cell foundation models (scFMs) preserve the regulatory information needed to explain cellular responses to perturbation? Existing evaluations measure predictive accuracy, which cannot establish which regulatory relationships their representations retain. We introduce CLAMP, a mechanistic probe that fits a gene regulatory network through a frozen encoder and evaluates the recovered network in gene space, where its interactions can be inspected and tested against independent biological references. We apply CLAMP to four scFMs (scGPT, scPRINT, Stack and Tahoe-x1) on four Perturb-seq datasets (Norman, Replogle K562, Replogle RPE1 and X-Atlas HCT116). Fitted directly in gene space, CLAMP recovers 70–89% of an empirical ceiling, predicts unseen two-gene interventions without combinatorial training, and recovers the known lineage programs of key regulators. Recovery through the scFMs generally falls below simple baselines such as PCA and random projections. The deficit concentrates in the part of each response that identifies its regulator: the recovered networks reproduce responses shared across interventions more readily than the differences between them. A layerwise analysis shows that this specificity is already reduced at the first transformer block and is not regained with depth, and that the encoders transmit lower-variance regulatory directions more weakly than leading ones. These results suggest pretraining objectives that reward distinguishing perturbations as a way to preserve regulatory specificity. Code will be released upon acceptance.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
single-cell foundation models, gene regulatory networks, mechanistic probing, perturbation modeling, representation evaluation, Perturb-seq

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dmCiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Single-cell foundation models has been shown to predict perturbation responses no better than simple baselines. This paper claims that the fault may lie with the predictor trained on top rather than with the model. Hence, the paper removes the predictor, fits a small gene-space regulatory network and uses the frozen scFM only to compute the loss. Control cells are edited to match the network's prediction, and their embedding shift is matched to that of real perturbed cells. The fitted networks are scored in gene space. Across four scFMs and four Perturb-seq datasets, networks fitted through the scFMs recover less than those fitted through PCA or random projections. The loss is mainly in what distinguishes one perturbation from another, and layer-wise readouts place it at the first block. The authors conclude that pretraining should reward telling perturbations apart.

Critical strengths
  • Existing benchmarks cannot tell whether a poor perturbation prediction comes from the scFM or from the predictor on top of it, so they cannot say what the scFM itself has learned. Here the frozen model serves only as the loss and the network stays in gene space. No trained predictor enters the comparison, and every encoder is scored on the same gene-space scale.

  • A negative result such as "scFMs recover less" means little unless the probe could have detected the signal through every encoder. Before any fitting, the authors check that the loss rewards the true answer through each encoder. They also add shuffled-label and random-weight controls that set a floor, and document all of it carefully.

  • Pretraining is expensive, and judging an scFM usually takes costly downstream benchmarks. The layer-wise readouts need no fitting, only the encoder's sensitivity at a few dozen cells. They could therefore be run during pretraining to check whether perturbation-specific differences survive.

Critical weaknesses
  • The paper seems to measures embedding geometry, not learned regulation. The probe must separate information an scFM has lost from information it keeps at a different scale, however, as built, it asks whether the pooled embedding stays sensitive, under a Euclidean loss, to perturbation-specific changes. Identity, PCA and random projections roughly preserve distances, so they score well almost by construction. Refitting on whitened embeddings would show whether the scFM gap survives.

  • The pipeline itself favors linear encoders, and a method that handicaps nonlinear models produces part of the scFM gap itself. Hence the gap cannot simply be blamed on pretraining. Choices tilt the comparison:

    • The cell edit matches only average gene levels, so even the true answer scores worse through the scFMs than with no encoder.
    • Fits through binned models use approximate gradients.
    • Settings were tuned on the no-encoder baseline.
  • Novelty is narrower than presented- the paper omits or underplays the work closest to its model and findings:

    • The network model follows CellBox (Yuan et al., 2021).
    • The headline result echoes earlier embedding-level evaluations in which PCA or simpler methods beat scFMs (Bendidi et al., 2024; Kedzierska et al., 2025).
    • The shared-versus-specific contrast is the central point of Systema (Viñas Torné et al., 2025).
    • CellBox, Bendidi et al. and Kedzierska et al. are not cited, and Systema is cited only in passing.
  • The paper's only test against independent biological references, and it supports the abstract's claim that the method "recovers the known lineage programs." Each regulator's predicted targets are fitted to its own response. The six regulators tested were chosen because their responses already match the references, so the check mostly confirms the fit. With only six regulators, and a shuffled control that still passes one, the differences between models are weak evidence. The abstract's claim is therefore overstated.

Questions for authors

see above.

4 comments

Thank you for taking the time to review our paper and for the constructive criticism. We’ve answered your questions below. Modifications in the revised paper are in blue. Let us know if you have further questions or comments!

Rescaling embedding directions. Thanks for this suggestion. We whitened each embedding with the covariance of the control cells, so that every direction has the same scale, and refitted all models on the Norman K562 dataset. In Table A we compare the original vs. the whitened results for held-out single (SGI) and unseen double interventions (DGI):

Table A. Recovery on SGI and unseen DGI on Norman, mean ± s.d. over three seeds, as in Table 1.

EncoderSGI, Table 1SGI, whitenedDGI, Table 1DGI, whitened
PCA (512)0.63 ± 0.030.57 ± 0.010.71 ± 0.030.69 ± 0.00
Random projection (512)0.46 ± 0.030.43 ± 0.030.55 ± 0.020.54 ± 0.02
scGPT0.27 ± 0.020.15 ± 0.010.35 ± 0.010.19 ± 0.01
scPRINT0.27 ± 0.020.16 ± 0.020.35 ± 0.020.22 ± 0.01
Stack-Large0.35 ± 0.010.07 ± 0.010.41 ± 0.010.09 ± 0.01
Tahoe-x10.15 ± 0.010.07 ± 0.000.23 ± 0.020.11 ± 0.01

After whitening, all four scFMs remain well below PCA and the random projection.

We posit that while whitening does correct the scale of the embedding directions, it can’t change how strongly a given encoder transmits a regulatory direction relative to the natural variation between cells. Whitening divides each direction by the spread of the control cells along them. In expression space the variance is 6x larger along the leading regulatory directions than along the weaker regulatory directions. Therefore, an encoder that preserves expression would transmit these weaker directions more strongly than leading ones after whitening. We check this by measuring the transmission of the weaker regulatory directions, as in Fig. 2f. In Table B we show that PCA and random projection transmit these 1.6 to 1.8 times more strongly. On the other hand for scFMs, we see whitening does raise the weaker directions, but only approx. to the level of the leading ones or even below (between 0.6 to 1.0). This measure is unchanged by any further linear rescaling of the embedding, so that would indicate attenuation is a property of the encoder.

Table B. Transmission of the weaker regulatory directions relative to the leading five, as in Fig. 2f.

EncoderOriginalWhitened
PCA (512)0.931.57
Random projection (512)0.981.78
scGPT0.610.97
scPRINT0.721.00
Stack-Large0.580.65
Tahoe-x10.600.83

In our conclusion we noted that reconstruction losses can favor high-variance co-expression programs. A model could therefore reconstruct expression well while attenuating low-variance, intervention-specific directions. These further results show that after whitening the gap persists and the weaker regulatory directions remain attenuated. This needs further study, and we have added these results to the appendix J and have referenced it in the text (lines 304-305 + 418-419).

Simple projections scoring well by construction. After whitening, PCA and the random projection no longer preserve distances in expression space: whitening rescales their directions relative to one another by up to 10-fold and 32-fold. They still recover far more than the scFMs (Table A), so their advantage does not come from preserving Euclidean distances.

Does the pipeline favor linear encoders? We asked ourselves the same question, and defined the linearized fits of Section 2.5 (Fig. 2a-d) to attempt to quantify what this error might be. On Norman K562 at every block of each model we fit the network twice: through the block itself and through its local linear approximation. We show the results in Table C. The difference between the two is our estimate of what the pipeline loses on a nonlinear encoder.

Table C. Recovery (DGI, Norman) through each scFM block and through its linear approximation: lowest to highest value across transformer blocks, as plotted in Fig. 2a–d. The difference is taken block by block.

EncoderOriginalLinearizedLinearized − original
scGPT0.23 to 0.420.31 to 0.42−0.02 to +0.12
scPRINT0.22 to 0.300.28 to 0.390.00 to +0.11
Stack-Large0.40 to 0.520.37 to 0.52−0.06 to +0.01
Tahoe-x10.12 to 0.260.27 to 0.44+0.11 to +0.24

This difference depends on the model and on the depth: very little for Stack-Large, up to 0.12 for scGPT and scPRINT, and 0.11 to 0.24 for Tahoe-x1 (as we note in lines 400–401). It does not account for the gap to Identity: no linearized fit exceeds 0.52, against 0.78 for Identity (Table 1). We do agree that there is a bias towards linear encoders, but we have tried to systematically quantify this and reduce it as much as possible. Reducing it further on nonlinear encoders remains as future work. We also agree that the gap should not be attributed to pretraining alone. The paper attributes it to the encoders’ learned mappings and input processing.

Furthermore, on the three choices you list:

  • Cell edit. Appendix E checks this before any fit: the true answer removes at least two thirds of the loss through every encoder (Table 2, p. 19).
  • Approximate gradients. scPRINT and Stack-Large read continuous values and need no approximation, and they show the same gap. For scGPT, the approximate gradient has a cosine of 0.67–0.90 with the exact one-count derivative (lines 1401–1403, p. 26).
  • Settings. We first compared learning rates on the scFMs directly, and the value that worked best on Identity generally worked best for them too. We therefore fixed it as a rule (Appendix K) rather than search per encoder: each fit backpropagates through the scFM, and every additional learning rate adds about 300 such fits across Table 1 + Fig. 2a–d. The transmission readouts of Fig. 2e–f involves no fit and no settings, and shows the same pattern.

Related work. Thank you for these references. In our introduction we write that benchmarks have shown that simple baselines match or beat scFMs for perturbation effect prediction (Ahlmann-Eltze, Wenteler, Svensson). In Related Work, we’ve now added Bendidi et al. (2024), who conclude similarly using PCA and scVI, as well as Kedzierska et al. (2025), whose work focused on frozen embeddings for cell-type clustering and batch integration. We’ve also expanded the sentence on Systema, which is certainly work we’ve read carefully and drawn inspiration from, and now mention it in the results section (lines 85 to 92 + Section 4.1 lines 388-390). We’ve added CellBox to the paragraph on dynamical systems (lines 77 to 79). In our opinion, we maintain the most similar work in terms of model is Bicycle, which we cite. Its steady state is the linear case of our Eq. 2, fitted to the same kind of data. CellBox differs in terms of training data, regulation and how a perturbation is encoded, but is certainly of the same family. CLAMP differs from both in how regulation is modelled (a saturating Hill response for each regulator) and in holding the perturbed gene at its measured level.

Novelty. The paper takes as its starting point that simple baselines outperform scFMs, which the benchmarks we cite have established. Similarly, we don’t present the dynamical system itself as new: CLAMP builds on a rich lineage of work in that field, as mentioned above. It also introduces two developments which, to our knowledge, are new to this field: implicit differentiation through the steady state, which enables it to scale tractably, and a network that stays in gene space, where it can be inspected, while the loss is computed in the scFM’s embedding space. Together these are what enable using it as a probe. The gradient comes from one linear solve per intervention, with no backpropagation through an ODE solver, so the training signal from a frozen encoder can reach a shared network of about 2,500 genes. The stated contributions (lines 95–107) are therefore a mechanistic probe fitted through a frozen encoder, the layerwise analysis, and what they enable us to analyse: which part of a response is less recoverable (the part that identifies its regulator), where this starts (the first transformer block) and how it happens (lower-variance directions are transmitted more weakly). This points to what could be changed to build better models: pretraining objectives that reward distinguishing perturbations, and the input encoding and first block, where specificity is already reduced. The transmission readouts need no fitting, so they could also be used to monitor this during pretraining.

Biological validation. A ChIP-seq reference describes a regulator in the cell type where it was measured, and most of our references come from other cell types (for example liver for HNF4A). We therefore first asked which references are visible in the K562 data itself, before fitting any network. Six of 19 were, and we tested the networks on those six.

We see that this step can look like selecting on the outcome, so we repeated the test without it, using the same fits. All 19 regulators with a reference are now included. The test is also stricter: a regulator’s recovered targets must match its own reference better than the targets of almost any other regulator do (top 3 of 102), which checks that the match is specific to that regulator (Table D). We’ve added this analysis to Appendix M and reference it in lines 476-478 of the main text.

Table D. Regulators whose recovered targets match their reference, out of 19; mean over three seeds. Chance is the count obtained when the recovered target lists are reassigned to random regulators.

EncoderRegulators validated (of 19)Expected by chance
Identity4.30.4
PCA4.70.5
scGPT2.70.6
scPRINT2.30.4
Stack-Large2.00.3
Tahoe-x110.3
Shuffled labels0.70.3
Random weights00.2

The shuffled control is at chance. Identity and PCA recover the most regulators. scGPT, scPRINT and Stack-Large recover fewer, but several times more than chance, and the same regulators in every seed (IRF1 and SPI1 for Stack-Large, IRF1 for scPRINT, CEBPA for scGPT).

We agree that the abstract should be more specific on this point. We’ve reworded it to “On Norman, it predicts unseen two-gene interventions without combinatorial training, and the targets it recovers for KLF1, SPI1 and IRF1 match published ChIP-seq and lineage gene sets for these regulators” (lines 19 to 23).

Dear Reviewer dmCi, as the discussion period closes today, we wanted to check whether our response addressed your concerns.

Following your suggestions, we:

    1. Refitted all encoders on whitened embeddings. The gap to PCA and random projections remains, and the weaker regulatory directions are still transmitted more weakly after rescaling.
    1. Quantified the bias towards linear encoders block by block using the linearised fits from Section 2.5, and showed that it does not account for the gap to the identity baseline.
    1. Repeated the biological validation on all 19 regulators with a reference, without the earlier pre-selection step and with a stricter test of specificity, and reworded the abstract to match.
    1. Added the related work you pointed us to.

If anything remains unclear we would be glad to clarify before the deadline. Thank you again for the constructive review.

Reviewer_aebcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

CLAMP evaluates single-cell foundation models by fitting a shared gene regulatory network through each frozen encoder. The network models steady states under sustained perturbations, while edited control cells connect its predictions to the encoder's input. Recovered networks are evaluated in gene space using perturbation responses, unseen combinations, and biological references. Across several encoders and datasets, simple projections generally support better recovery. Layerwise analyses suggest that the models preserve shared response patterns more readily than regulator-specific differences.

Critical strengths
  • The motivation is clear and important. Using an explicit regulatory network as the probe adds a useful perspective beyond prediction accuracy. The response, retrieval, and biological-reference evaluations give complementary evidence about what the probe recovers.
  • The controls make the negative results informative. Identity, PCA, random projections, and shuffled responses help interpret the recovery gap. The layerwise sensitivity analysis also connects that gap to weaker transmission of regulator-specific directions.
Critical weaknesses
  • The conclusions about regulatory knowledge are broader than the measurement. CLAMP tests a particular cell embedding with a squared-distance loss. The direction-dependent attenuation in Figure 1 could make information harder for this probe to use without removing it from the model. A check with a reweighted embedding loss or a different readout would help distinguish these possibilities. The current evidence supports a limitation of this recovery procedure more directly than a general loss of regulatory information.
Questions for authors
  1. Does the recovery gap persist after whitening or reweighting embedding directions to reduce their scale imbalance?
  2. Have you checked whether a gene-level readout preserves the regulator-specific information that is weak in the cell embedding?
3 comments

Thank you for taking the time to review our paper and for the constructive criticism. We’ve answered your questions below. Modifications in the revised paper are in blue. Let us know if you have further questions or comments!

Rescaling embedding directions. Thanks for this suggestion. We whitened each embedding with the covariance of the control cells, so that every direction has the same scale, and refitted all models on the Norman K562 dataset. In Table A we compare the original vs. the whitened results for held-out single (SGI) and unseen double interventions (DGI):

Table A. Recovery on SGI and unseen DGI on Norman, mean ± s.d. over three seeds, as in Table 1.

EncoderSGI, Table 1SGI, whitenedDGI, Table 1DGI, whitened
PCA (512)0.63 ± 0.030.57 ± 0.010.71 ± 0.030.69 ± 0.00
Random projection (512)0.46 ± 0.030.43 ± 0.030.55 ± 0.020.54 ± 0.02
scGPT0.27 ± 0.020.15 ± 0.010.35 ± 0.010.19 ± 0.01
scPRINT0.27 ± 0.020.16 ± 0.020.35 ± 0.020.22 ± 0.01
Stack-Large0.35 ± 0.010.07 ± 0.010.41 ± 0.010.09 ± 0.01
Tahoe-x10.15 ± 0.010.07 ± 0.000.23 ± 0.020.11 ± 0.01

After whitening, all four scFMs remain well below PCA and the random projection.

We posit that while whitening does correct the scale of the embedding directions, it can’t change how strongly a given encoder transmits a regulatory direction relative to the natural variation between cells. Whitening divides each direction by the spread of the control cells along them. In expression space the variance is 6x larger along the leading regulatory directions than along the weaker regulatory directions. Therefore, an encoder that preserves expression would transmit these weaker directions more strongly than leading ones after whitening. We check this by measuring the transmission of the weaker regulatory directions, as in Fig. 2f. In Table B we show that PCA and random projection transmit these 1.6 to 1.8 times more strongly. On the other hand for scFMs, we see whitening does raise the weaker directions, but only approx. to the level of the leading ones or even below (between 0.6 to 1.0). This measure is unchanged by any further linear rescaling of the embedding, so that would indicate attenuation is a property of the encoder.

Table B. Transmission of the weaker regulatory directions relative to the leading five, as in Fig. 2f.

EncoderOriginalWhitened
PCA (512)0.931.57
Random projection (512)0.981.78
scGPT0.610.97
scPRINT0.721.00
Stack-Large0.580.65
Tahoe-x10.600.83

Scope of the conclusion. In our conclusion we noted that reconstruction losses can favor high-variance co-expression programs. A model could therefore reconstruct expression well while attenuating low-variance, intervention-specific directions. These further results show that after whitening the gap persists and the weaker regulatory directions remain attenuated. This needs further study, and we have added these results to the appendix J and have referenced it in the text (lines 304-305 + 418-419).

Gene-level readouts. We haven’t tackled gene-level readouts, although we agree they would be a natural and logical next step. There are several technical issues that stop us from implementing this now:

  • Gene tokens aren’t always a matched readout. CLAMP compares the predicted vs the observed embedding shift. The shift is the difference between the embeddings of different cells, which must therefore be matched: same length, with entries corresponding to the same feature. Cell embeddings are matched by construction because every cell gets the same set of features. Gene tokens are not matched: in scGPT and Tahoe-x1 a cell has tokens only for the genes which are “on”, so two cells can have tokens for different genes. Therefore we would have to rethink the loss and the readout to enable this analysis.
  • Not available for all four models. Stack has no per-gene tokens, so we would also have to devise a different strategy here.
  • New controls. Because this is a different question, we would have to devise new controls which would enable us to determine if we trust the results we are seeing or not. For example, because a gene token contains its own value, recovery might be high without reflecting learned regulation.

I appreciate the authors' rebuttal. I think it indeed resolves my concerns. I agree with Reviewer gHtt that well known models for GRN inference should be systematically compared to the proposed method so the conclusion of whether the proposed method's capability on identifying regulatory target is solid can be clear. I read the response to Reviewer gHtt, however, I think the benchmark is not detailed enough, making me hesitated to increase my score further.

Thanks, we're glad we were able to address your concerns!

Regarding the comparison with GRN inference methods, we think we did not explain it clearly enough in the reply to Reviewer gHtt, so we lay it out more fully here, and have expanded Appendix M.9 (p. 35-36) and the Results section (lines 470-474) to match.

GRN inference. GRN inference methods fit, for each gene, a model predicting its expression from the expression of the transcription factors (TF) across cells, so an edge means that a TF's level correlates with the gene’s level in the population. Such co-expression can arise from regulation, but also from shared cell state or technical effects, because no gene is ever intervened upon. CLAMP instead fits the change in expression when a regulator is actually perturbed, so an edge is an interventional effect, and it exists only for the perturbed regulators. The two approaches therefore use different evidence, and because no ground-truth network exists for these cells, neither can simply be scored against “the truth”. To compare their outputs in a way that answers whether each method identifies the right targets, we took from each method its 25 strongest targets for every perturbed regulator and scored those lists on the same two tests: how many of the 25 targets are among the genes whose expression actually changes when that regulator is perturbed, and whether the 25 targets are enriched for the regulator’s known targets from independently published ChIP-seq or lineage gene sets.

We ran SimiC, GRNBoost2 and GENIE3, three methods the GRN inference field treats as references. In the BEELINE benchmark of single-cell GRN inference (Pratapa et al. 2020), GENIE3 and GRNBoost2 are among the three best-performing methods on experimental data, and SimiC was suggested by reviewer gHtt. We ran them on the same cells and genes as CLAMP, giving them the control and the perturbed cells together, took each method’s 25 strongest targets per regulator, and scored all four on the same two tests, each with its own chance baseline. Test 1: how many of the 25 targets change when the regulator is perturbed. Test 2, with references no method was fitted to: of 19 regulators with a published ChIP-seq or lineage set, how many have targets that match it.

Table C. Mean ± s.d. over three seeds for CLAMP and GRNBoost2, one seed for SimiC and GENIE3. Chance for Test 1 is a regulator's list scored against the other regulators' responses. Chance for Test 2 is the count obtained when each regulator is scored with another regulator's list, averaged over 20,000 random reassignments. Details in Appendix M.9.

MethodTest 1: responding targets, of 25ChanceTest 2: regulators validated, of 19Chance
CLAMP6.7 ± 0.12.34.3 ± 0.60.4
SimiC1.50.530.1
GRNBoost21.1 ± 0.00.12.3 ± 0.60.1
GENIE30.40.0520.05

CLAMP does better than the three GRN methods on both tests. Test 1 is expected to favor CLAMP, because the perturbation responses are what it is fitted to. Test 2 does not favor it, since none of the methods were fitted to the ChIP-seq or lineage sets, and CLAMP still validates more regulators (4.3 of 19 against 3 for SimiC) than GRN methods. The GRN methods perform as per published benchmarks on experimental single-cell data (BEELINE, Pratapa et al. 2020). One reason they underperform compared to CLAMP is easy to see in our data: the genes they rank as top targets are detected in only 0.3 to 7% of control cells, while the genes that actually respond to a perturbation are detected in 78%. Co-expression is therefore weak evidence for genes that are rarely detected, and perturbation data supplies the signal that co-expression cannot. We take this as adequate evidence that the network CLAMP recovers in gene space identifies real regulatory targets and is a reasonable reference for the scFM results. Protocol, seeds and p-values are given in Appendix M.9.

Reviewer_gHttOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose CLAMP, a framework to evaluate the capacities of single cell foundation models to capture regulatory mechanisms. The authors also propose a very comprehensive framework to evaluate these capacities through the recovery of GRNs. The authors compared 4 very recent scFMs and evaluate their embedded space. They also use several baselines, and show that the baselines capture more accurate gene-gene relationships. Finally, the authors propose 3 needed changes in training new scFM models to improve their reliability.

Critical strengths

The authors propose a benchmark that addresses a critical need for ensuring that single-cell foundation models (scFMs) actually generate an embedding space that captures proper biology. This is a key feature, as current scFM models seems to be aiming mostly at improving predicted performance, while ignoring the capability of the inferred embedded space (the freeze Encoder) to represent biologically-meaningful latent vectors.

Critical weaknesses

The authors missed several important scFMs, such as Geneformer, Geneformer2, TranscriptFormer, scFoundation, UCE, among others. As a benchmarking paper (which also provides a novel way of assessing scFM models) it is of critical importance to have a comprehensive evaluation of current FMs.

The authors should compare their proposed way of inferring the GRNs, with GRNs inferred from dedicated models for such task, as SERGIO, SCENIC+, SIMIC. That is, if I infer a GRN associated with control/perturbed cells, and then compared with the one inferred by CLAMP, how do they differ?

Also, the paper is quite difficult to follow. The inference of parameters θ\theta, from the freezer encoders is not clearly described. A figure on the CLAMP architecture and the different layered analysis would help.

Questions for authors

How do the proposed benchmark adds over the already published benchmarks? While there is a short sentence in the introduction, the added value of current benchmark should me more clearly assessed as otherwise it makes this work quite incremental.

How well the inferred GRNs compare against well known models for GRN inference? Could CLAMP be based on pre-defined GRNs, or at least, leverage them in some form?

What about using (sc)ATAC-seq data, would this type of additional omics layer help/improve the benchmarking capabilities?

4 comments

Thank you for taking the time to review our paper and for offering constructive criticism. We’ve answered your questions below and where possible took your feedback on board. Modifications in the revised paper are in blue. Let us know if you have further questions or comments!

CLAMP pipeline. Thank you for this suggestion. We’ve added a full pipeline figure to the main text (Figure 1, page 3), with a step-by-step explanation which we hope will make the paper easier to follow. In short, the GRN is fitted through each frozen scFM, with no predictor trained on top, so that the fit is guided by the scFM alone. This required three technical developments: backpropagation through the frozen scFM and its input processing; implicit differentiation through the network's steady state, which makes the fit tractable; and a network that stays in gene space, where it can be inspected, while the loss is computed in the scFM's embedding space. To our knowledge, the last two are new to this field.

Added value over published benchmarks. We’ve expanded the introduction to make the added value explicit (lines 45 to 55): benchmarks of predictive accuracy are descriptive, and the aim of CLAMP is to move beyond description and begin to explain why scFMs underperform. This is our reasoning: fundamentally, we developed CLAMP as a method enabling us to probe scFMs, rather than as a benchmark of all current models. As a community working on perturbation effect prediction, we know that scFMs underperform at this task. To our knowledge, there have been few efforts dedicated to finding out and codifying why this is happening. CLAMP is a step in that direction: it enables us to identify where and how scFMs are falling short, to infer why this might be happening and it gives us mechanistic tools to compare inferred GRNs across models in gene space. Each component of the pipeline was designed and purpose built to enable this comparison (see the CLAMP pipeline section above). Ultimately, the aim is to build better foundation models that do codify regulatory biological relationships, building on the insight CLAMP can give us.

Coverage of scFMs. We agree with you that the models you list are important and that broader coverage is valuable. Two considerations guided our model choice:

  1. Method constraint. CLAMP fits the GRN through the frozen scFM, which requires gradients of the embedding with respect to gene expression. This is both a strength and a limitation: the scFM itself guides the fit, but not every model provides these gradients. scPRINT and Stack take numerical expression input, and hence their gradient is exact. scGPT and Tahoe-x1 bin expression, so we built and validated a relaxation for the gradient, checking it against exact differences (see Appendix J). For Geneformer v1/v2 genes are input as ranks, and for UCE genes are randomly sampled according to their expression, so neither have an exact gradient. Both would require a new relaxation be built and validated.
  2. Design choice. Rather than provide comprehensive coverage, we chose models that allow closely matched comparisons of specific scFM design choices. For example, scGPT and Tahoe-x1 both bin expression, read expressed genes only and are trained by masking: their main difference is that Tahoe-x1 was trained on perturbation data and scGPT was not. We didn’t explain this fully in the text, so we’ve now added this explanation in the main paper (lines 294 to 298). The table below shows the design choices made by each scFM tested in the paper:
scGPTTahoe-x1scPRINTStack
Input valuesBinnedBinnedNumericalNumerical
Genes in inputExpressed genes onlyExpressed genes onlyAllAll
Training taskMaskingMaskingDenoisingMasking
Perturbation data in trainingNoYesNoNo
Reads at onceOne cellOne cellOne cellMany cells

A logical extension would be adding scFoundation, because it combines numerical input with expressed genes only - which is a combination not currently covered by the other models. However, we would rather not rush adding this during the rebuttal period, since each new model requires careful validation work (see Appendices E and J) before we would feel comfortable publishing any results.

Comparison with GRN inference methods. Thank you for this suggestion. We ran this comparison on Norman and have added it to the paper (lines 323-325 + 467-468 + Appendix M). SERGIO simulates expression from a given network rather than inferring one, and SCENIC+ needs chromatin accessibility data, which these Perturb-seq datasets don’t include. We therefore ran SimiC, as you suggested, together with GRNBoost2 (the inference step of SCENIC) and GENIE3, each on control and perturbed cells.

To answer how they differ: the networks are largely different. For a given regulator, CLAMP and a GRN method share only 1–2 of their 25 strongest targets. We then tested each method’s top 25 predicted targets per regulator in two ways: (1) how many of them change in expression when that regulator is perturbed, and (2) whether they match the regulator’s known targets from independent ChIP-seq and lineage gene sets.

MethodTargets that change under perturbationRegulators whose known targets are matched
CLAMP6.7 of 25KLF1, SPI1, IRF1, CEBPA
SimiC1.5 of 25KLF1, SPI1, IRF1
GRNBoost21.1 of 25KLF1, IRF1
GENIE30.4 of 25CEBPA, CEBPE
Random targets0.2 of 25none

The two approaches use different evidence: GRN methods ask which genes vary together across cells, while CLAMP is fitted to how genes change under perturbation. The first column therefore mainly shows what perturbation data adds. The second column uses references that no method was fitted to, and there CLAMP and SimiC recover the same three well-characterised programs.

Pre-defined GRNs and (sc)ATAC-seq. Yes, CLAMP can use a pre-defined GRN directly, by fitting only the links it contains. The table suggests that a GRN inferred from expression alone would be a weak starting point, since few of its targets respond to perturbation. Chromatin accessibility looks more promising to us: it indicates which regulator–target links are plausible in a given cell type, so it could serve both as a prior for the fit and as an independent reference to check recovered targets for more regulators than ChIP-seq covers. The datasets in this paper are RNA-only, so we can’t test this hypothesis on them, but it would be promising future work.

AuthorsOpenReview ↗

Update to the Comparison with GRN inference methods.

Dear Reviewer gHtt, following a comment from Reviewer aebc we think our explanation on the GRN inference comparison was not clear or detailed enough, so we set it out again more fully below. Appendix M.9 (p. 35-36) and the Results section (lines 470-474) have been expanded to match.

GRN inference. GRN inference methods fit, for each gene, a model predicting its expression from the expression of the transcription factors (TF) across cells, so an edge means that a TF's level correlates with the gene’s level in the population. Such co-expression can arise from regulation, but also from shared cell state or technical effects, because no gene is ever intervened upon. CLAMP instead fits the change in expression when a regulator is actually perturbed, so an edge is an interventional effect, and it exists only for the perturbed regulators. The two approaches therefore use different evidence, and because no ground-truth network exists for these cells, neither can simply be scored against “the truth”. To compare their outputs in a way that answers whether each method identifies the right targets, we took from each method its 25 strongest targets for every perturbed regulator and scored those lists on the same two tests: how many of the 25 targets are among the genes whose expression actually changes when that regulator is perturbed, and whether the 25 targets are enriched for the regulator’s known targets from independently published ChIP-seq or lineage gene sets.

We ran SimiC, GRNBoost2 and GENIE3, three methods the GRN inference field treats as references. In the BEELINE benchmark of single-cell GRN inference (Pratapa et al. 2020), GENIE3 and GRNBoost2 are among the three best-performing methods on experimental data, and SimiC was suggested by reviewer gHtt. We ran them on the same cells and genes as CLAMP, giving them the control and the perturbed cells together, took each method’s 25 strongest targets per regulator, and scored all four on the same two tests, each with its own chance baseline. Test 1: how many of the 25 targets change when the regulator is perturbed. Test 2, with references no method was fitted to: of 19 regulators with a published ChIP-seq or lineage set, how many have targets that match it.

Table C. Mean ± s.d. over three seeds for CLAMP and GRNBoost2, one seed for SimiC and GENIE3. Chance for Test 1 is a regulator's list scored against the other regulators' responses. Chance for Test 2 is the count obtained when each regulator is scored with another regulator's list, averaged over 20,000 random reassignments. Details in Appendix M.9.

MethodTest 1: responding targets, of 25ChanceTest 2: regulators validated, of 19Chance
CLAMP6.7 ± 0.12.34.3 ± 0.60.4
SimiC1.50.530.1
GRNBoost21.1 ± 0.00.12.3 ± 0.60.1
GENIE30.40.0520.05

CLAMP does better than the three GRN methods on both tests. Test 1 is expected to favor CLAMP, because the perturbation responses are what it is fitted to. Test 2 does not favor it, since none of the methods were fitted to the ChIP-seq or lineage sets, and CLAMP still validates more regulators (4.3 of 19 against 3 for SimiC) than GRN methods. The GRN methods perform as per published benchmarks on experimental single-cell data (BEELINE, Pratapa et al. 2020). One reason they underperform compared to CLAMP is easy to see in our data: the genes they rank as top targets are detected in only 0.3 to 7% of control cells, while the genes that actually respond to a perturbation are detected in 78%. Co-expression is therefore weak evidence for genes that are rarely detected, and perturbation data supplies the signal that co-expression cannot. We take this as adequate evidence that the network CLAMP recovers in gene space identifies real regulatory targets and is a reasonable reference for the scFM results. Protocol, seeds and p-values are given in Appendix M.9.

Dear Reviewer gHtt, as the discussion period closes today, we wanted to check whether our response addressed your concerns.

Following your suggestions, we:

    1. Compared CLAMP against SimiC, GRNBoost2 and GENIE3 on Norman, scoring each method on perturbation response and on independent ChIP-seq and lineage references.
    1. Added a full pipeline figure with a step-by-step explanation of how the network is fitted through the frozen encoder.
    1. Rewrote the introduction to make explicit CLAMP’s added value over predictive benchmarks.

If anything remains unclear, we would be glad to clarify before the deadline. Thank you again for the constructive review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.