- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Single-cell foundation models has been shown to predict perturbation responses no better than simple baselines. This paper claims that the fault may lie with the predictor trained on top rather than with the model. Hence, the paper removes the predictor, fits a small gene-space regulatory network and uses the frozen scFM only to compute the loss. Control cells are edited to match the network's prediction, and their embedding shift is matched to that of real perturbed cells. The fitted networks are scored in gene space. Across four scFMs and four Perturb-seq datasets, networks fitted through the scFMs recover less than those fitted through PCA or random projections. The loss is mainly in what distinguishes one perturbation from another, and layer-wise readouts place it at the first block. The authors conclude that pretraining should reward telling perturbations apart.
Critical strengths
-
Existing benchmarks cannot tell whether a poor perturbation prediction comes from the scFM or from the predictor on top of it, so they cannot say what the scFM itself has learned. Here the frozen model serves only as the loss and the network stays in gene space. No trained predictor enters the comparison, and every encoder is scored on the same gene-space scale.
-
A negative result such as "scFMs recover less" means little unless the probe could have detected the signal through every encoder. Before any fitting, the authors check that the loss rewards the true answer through each encoder. They also add shuffled-label and random-weight controls that set a floor, and document all of it carefully.
-
Pretraining is expensive, and judging an scFM usually takes costly downstream benchmarks. The layer-wise readouts need no fitting, only the encoder's sensitivity at a few dozen cells. They could therefore be run during pretraining to check whether perturbation-specific differences survive.
Critical weaknesses
-
The paper seems to measures embedding geometry, not learned regulation. The probe must separate information an scFM has lost from information it keeps at a different scale, however, as built, it asks whether the pooled embedding stays sensitive, under a Euclidean loss, to perturbation-specific changes. Identity, PCA and random projections roughly preserve distances, so they score well almost by construction. Refitting on whitened embeddings would show whether the scFM gap survives.
-
The pipeline itself favors linear encoders, and a method that handicaps nonlinear models produces part of the scFM gap itself. Hence the gap cannot simply be blamed on pretraining. Choices tilt the comparison:
- The cell edit matches only average gene levels, so even the true answer scores worse through the scFMs than with no encoder.
- Fits through binned models use approximate gradients.
- Settings were tuned on the no-encoder baseline.
-
Novelty is narrower than presented- the paper omits or underplays the work closest to its model and findings:
- The network model follows CellBox (Yuan et al., 2021).
- The headline result echoes earlier embedding-level evaluations in which PCA or simpler methods beat scFMs (Bendidi et al., 2024; Kedzierska et al., 2025).
- The shared-versus-specific contrast is the central point of Systema (Viñas Torné et al., 2025).
- CellBox, Bendidi et al. and Kedzierska et al. are not cited, and Systema is cited only in passing.
-
The paper's only test against independent biological references, and it supports the abstract's claim that the method "recovers the known lineage programs." Each regulator's predicted targets are fitted to its own response. The six regulators tested were chosen because their responses already match the references, so the check mostly confirms the fit. With only six regulators, and a shuffled control that still passes one, the differences between models are weak evidence. The abstract's claim is therefore overstated.
Questions for authors
see above.
4 comments
Does the pipeline favor linear encoders? We asked ourselves the same question, and defined the linearized fits of Section 2.5 (Fig. 2a-d) to attempt to quantify what this error might be. On Norman K562 at every block of each model we fit the network twice: through the block itself and through its local linear approximation. We show the results in Table C. The difference between the two is our estimate of what the pipeline loses on a nonlinear encoder.
Table C. Recovery (DGI, Norman) through each scFM block and through its linear approximation: lowest to highest value across transformer blocks, as plotted in Fig. 2a–d. The difference is taken block by block.
| Encoder | Original | Linearized | Linearized − original |
|---|---|---|---|
| scGPT | 0.23 to 0.42 | 0.31 to 0.42 | −0.02 to +0.12 |
| scPRINT | 0.22 to 0.30 | 0.28 to 0.39 | 0.00 to +0.11 |
| Stack-Large | 0.40 to 0.52 | 0.37 to 0.52 | −0.06 to +0.01 |
| Tahoe-x1 | 0.12 to 0.26 | 0.27 to 0.44 | +0.11 to +0.24 |
This difference depends on the model and on the depth: very little for Stack-Large, up to 0.12 for scGPT and scPRINT, and 0.11 to 0.24 for Tahoe-x1 (as we note in lines 400–401). It does not account for the gap to Identity: no linearized fit exceeds 0.52, against 0.78 for Identity (Table 1). We do agree that there is a bias towards linear encoders, but we have tried to systematically quantify this and reduce it as much as possible. Reducing it further on nonlinear encoders remains as future work. We also agree that the gap should not be attributed to pretraining alone. The paper attributes it to the encoders’ learned mappings and input processing.
Furthermore, on the three choices you list:
- Cell edit. Appendix E checks this before any fit: the true answer removes at least two thirds of the loss through every encoder (Table 2, p. 19).
- Approximate gradients. scPRINT and Stack-Large read continuous values and need no approximation, and they show the same gap. For scGPT, the approximate gradient has a cosine of 0.67–0.90 with the exact one-count derivative (lines 1401–1403, p. 26).
- Settings. We first compared learning rates on the scFMs directly, and the value that worked best on Identity generally worked best for them too. We therefore fixed it as a rule (Appendix K) rather than search per encoder: each fit backpropagates through the scFM, and every additional learning rate adds about 300 such fits across Table 1 + Fig. 2a–d. The transmission readouts of Fig. 2e–f involves no fit and no settings, and shows the same pattern.
Related work. Thank you for these references. In our introduction we write that benchmarks have shown that simple baselines match or beat scFMs for perturbation effect prediction (Ahlmann-Eltze, Wenteler, Svensson). In Related Work, we’ve now added Bendidi et al. (2024), who conclude similarly using PCA and scVI, as well as Kedzierska et al. (2025), whose work focused on frozen embeddings for cell-type clustering and batch integration. We’ve also expanded the sentence on Systema, which is certainly work we’ve read carefully and drawn inspiration from, and now mention it in the results section (lines 85 to 92 + Section 4.1 lines 388-390). We’ve added CellBox to the paragraph on dynamical systems (lines 77 to 79). In our opinion, we maintain the most similar work in terms of model is Bicycle, which we cite. Its steady state is the linear case of our Eq. 2, fitted to the same kind of data. CellBox differs in terms of training data, regulation and how a perturbation is encoded, but is certainly of the same family. CLAMP differs from both in how regulation is modelled (a saturating Hill response for each regulator) and in holding the perturbed gene at its measured level.
Novelty. The paper takes as its starting point that simple baselines outperform scFMs, which the benchmarks we cite have established. Similarly, we don’t present the dynamical system itself as new: CLAMP builds on a rich lineage of work in that field, as mentioned above. It also introduces two developments which, to our knowledge, are new to this field: implicit differentiation through the steady state, which enables it to scale tractably, and a network that stays in gene space, where it can be inspected, while the loss is computed in the scFM’s embedding space. Together these are what enable using it as a probe. The gradient comes from one linear solve per intervention, with no backpropagation through an ODE solver, so the training signal from a frozen encoder can reach a shared network of about 2,500 genes. The stated contributions (lines 95–107) are therefore a mechanistic probe fitted through a frozen encoder, the layerwise analysis, and what they enable us to analyse: which part of a response is less recoverable (the part that identifies its regulator), where this starts (the first transformer block) and how it happens (lower-variance directions are transmitted more weakly). This points to what could be changed to build better models: pretraining objectives that reward distinguishing perturbations, and the input encoding and first block, where specificity is already reduced. The transmission readouts need no fitting, so they could also be used to monitor this during pretraining.
Biological validation. A ChIP-seq reference describes a regulator in the cell type where it was measured, and most of our references come from other cell types (for example liver for HNF4A). We therefore first asked which references are visible in the K562 data itself, before fitting any network. Six of 19 were, and we tested the networks on those six.
We see that this step can look like selecting on the outcome, so we repeated the test without it, using the same fits. All 19 regulators with a reference are now included. The test is also stricter: a regulator’s recovered targets must match its own reference better than the targets of almost any other regulator do (top 3 of 102), which checks that the match is specific to that regulator (Table D). We’ve added this analysis to Appendix M and reference it in lines 476-478 of the main text.
Table D. Regulators whose recovered targets match their reference, out of 19; mean over three seeds. Chance is the count obtained when the recovered target lists are reassigned to random regulators.
| Encoder | Regulators validated (of 19) | Expected by chance |
|---|---|---|
| Identity | 4.3 | 0.4 |
| PCA | 4.7 | 0.5 |
| scGPT | 2.7 | 0.6 |
| scPRINT | 2.3 | 0.4 |
| Stack-Large | 2.0 | 0.3 |
| Tahoe-x1 | 1 | 0.3 |
| Shuffled labels | 0.7 | 0.3 |
| Random weights | 0 | 0.2 |
The shuffled control is at chance. Identity and PCA recover the most regulators. scGPT, scPRINT and Stack-Large recover fewer, but several times more than chance, and the same regulators in every seed (IRF1 and SPI1 for Stack-Large, IRF1 for scPRINT, CEBPA for scGPT).
We agree that the abstract should be more specific on this point. We’ve reworded it to “On Norman, it predicts unseen two-gene interventions without combinatorial training, and the targets it recovers for KLF1, SPI1 and IRF1 match published ChIP-seq and lineage gene sets for these regulators” (lines 19 to 23).
Dear Reviewer dmCi, as the discussion period closes today, we wanted to check whether our response addressed your concerns.
Following your suggestions, we:
-
- Refitted all encoders on whitened embeddings. The gap to PCA and random projections remains, and the weaker regulatory directions are still transmitted more weakly after rescaling.
-
- Quantified the bias towards linear encoders block by block using the linearised fits from Section 2.5, and showed that it does not account for the gap to the identity baseline.
-
- Repeated the biological validation on all 19 regulators with a reference, without the earlier pre-selection step and with a stricter test of specificity, and reworded the abstract to match.
-
- Added the related work you pointed us to.
If anything remains unclear we would be glad to clarify before the deadline. Thank you again for the constructive review.
Thank you for taking the time to review our paper and for the constructive criticism. We’ve answered your questions below. Modifications in the revised paper are in blue. Let us know if you have further questions or comments!
Rescaling embedding directions. Thanks for this suggestion. We whitened each embedding with the covariance of the control cells, so that every direction has the same scale, and refitted all models on the Norman K562 dataset. In Table A we compare the original vs. the whitened results for held-out single (SGI) and unseen double interventions (DGI):
Table A. Recovery on SGI and unseen DGI on Norman, mean ± s.d. over three seeds, as in Table 1.
After whitening, all four scFMs remain well below PCA and the random projection.
We posit that while whitening does correct the scale of the embedding directions, it can’t change how strongly a given encoder transmits a regulatory direction relative to the natural variation between cells. Whitening divides each direction by the spread of the control cells along them. In expression space the variance is 6x larger along the leading regulatory directions than along the weaker regulatory directions. Therefore, an encoder that preserves expression would transmit these weaker directions more strongly than leading ones after whitening. We check this by measuring the transmission of the weaker regulatory directions, as in Fig. 2f. In Table B we show that PCA and random projection transmit these 1.6 to 1.8 times more strongly. On the other hand for scFMs, we see whitening does raise the weaker directions, but only approx. to the level of the leading ones or even below (between 0.6 to 1.0). This measure is unchanged by any further linear rescaling of the embedding, so that would indicate attenuation is a property of the encoder.
Table B. Transmission of the weaker regulatory directions relative to the leading five, as in Fig. 2f.
In our conclusion we noted that reconstruction losses can favor high-variance co-expression programs. A model could therefore reconstruct expression well while attenuating low-variance, intervention-specific directions. These further results show that after whitening the gap persists and the weaker regulatory directions remain attenuated. This needs further study, and we have added these results to the appendix J and have referenced it in the text (lines 304-305 + 418-419).
Simple projections scoring well by construction. After whitening, PCA and the random projection no longer preserve distances in expression space: whitening rescales their directions relative to one another by up to 10-fold and 32-fold. They still recover far more than the scFMs (Table A), so their advantage does not come from preserving Euclidean distances.