- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
scaffold predicts expression changes for gene perturbations that were never measured. Each gene gets a small learned vector, and genes with overlapping Gene Ontology annotations are encouraged to have similar representations. A shared linear map turns that representation together with the control-cell state into a predicted response. On nine datasets from scPerturBench it ranks first on seven and third on the remaining two, ahead of substantially more complex models.
Critical strengths
I like this paper. Seeing a small linear model come out on top of a public benchmark is useful, especially when much of the field is moving toward increasingly large pretrained models. I appreciated that the authors reproduced the benchmark's published rankings before inserting their own method. The comparison of GO, STRING, coexpression, and different GO neighborhood sizes is also practically useful.
Critical weaknesses
Novelty is limited. Graph-smoothing perturbation representations followed by a linear readout is conceptually simple, and related methods such as GEARS already use GO-derived perturbation graphs. What worries me more is the neighbor-averaging baseline in the supplement. It simply averages responses from GO-connected training perturbations. scaffold has better mean PCC-delta, but neighbor averaging actually has lower mean MSE. Since this very simple comparator captures a substantial part of the signal, I think it belongs in the main comparison. It would help distinguish the value of the biological prior itself from the value added by the learned model.
The other issue is potential contamination through the GO resource. For the relationship analyses in Figures 2 and 4, the authors remove GO annotation records whose cited source is the study under analysis. The benchmark predictor, however, uses the supplied GO resource without this filtering. This is not necessarily leakage, but it creates a temporal or post-study contamination risk: if annotations were added based on results from the same perturbation screens being predicted, they could encode information downstream of the evaluation data.
Questions for authors
-
Does the main benchmark ranking hold when using the same study-filtered GO construction used in the relationship analyses, or using a GO release that predates each perturbation dataset?
-
Where does the GO neighbor-averaging baseline rank if it is evaluated under the complete benchmark procedure rather than reported only as a mean-only comparator?
-
How does scaffold perform specifically on targets with no direct training-set neighbor in the GO graph? The paper retains these targets in the evaluation, and this seems like the most informative regime for testing whether the learned multi-hop representation adds value beyond nearest-neighbor transfer.
2 comments
Response to Reviewer eMJR (2/2): Questions 2 and 3
Question 2. "Where does the GO neighbor-averaging baseline rank if it is evaluated under the complete benchmark procedure rather than reported only as a mean-only comparator?"
Response. Under the complete benchmark procedure, neighbor averaging ranks second, with a mean rank of 1.89; scaffold has 1.44, and both are ahead of the 15 released methods and TxPert-public. The strength of neighbor averaging shows that GO relationships are themselves strongly predictive; scaffold improves further on top of it. These are the two parts the reviewer wishes to separate.
The 1.44 here is the rank of scaffold's predicted mean in this extended comparison; it is not the same comparison as the 1.44 in the paper's 17-method table, and the two values coincide by chance. Three of the six metrics compare cell distributions, and neighbor averaging produces no cells, so cells are generated in the same way for both methods: each gene of the control cells is shifted as a whole so that the mean of the cells equals the predicted mean. We apply the same procedure to scaffold, starting from its predicted mean, instead of taking the cells that scaffold outputs directly. Otherwise the three distribution metrics would compare two ways of generating cells, not the predictions of the two methods. We therefore report this as an extended comparison.
Over all nine datasets, scaffold's PCC-delta is 0.031 [0.016, 0.049] higher than that of neighbor averaging. Table 1 separates the two parts on K562 and RPE1, the two datasets with the most test genes: the step from the training mean to neighbor averaging is the gain from GO relationships, and the step from neighbor averaging to scaffold is the gain from the learned model.
| Table 1: PCC-delta | Training mean | Neighbor averaging | scaffold |
|---|---|---|---|
| K562 | 0.429 | 0.534 | 0.573 |
| RPE1 | 0.559 | 0.597 | 0.618 |
The reviewer notes that neighbor averaging has lower MSE. This holds only in the easier setting. In a harder one, where all genes of a complex or functional module are held out together, the difference reverses: scaffold has lower MSE, and its lead in PCC-delta grows as well (Table 2). The learned model shows its advantage where the task is harder. The test was run on K562 and RPE1.
| Table 2: scaffold minus neighbor averaging | PCC-delta | MSE | Reading |
|---|---|---|---|
| Random splits | +0.025 [0.018, 0.033] | +0.0032 [0.0022, 0.0043] | PCC-delta: scaffold higher; MSE: neighbor averaging lower |
| Whole GO-term gene sets held out | +0.082 [0.072, 0.092] | −0.0030 [−0.0048, −0.0013] | scaffold better on both |
| Whole CORUM complexes held out | +0.110 [0.097, 0.122] | −0.0057 [−0.0078, −0.0036] | scaffold better on both |
Question 3. "How does scaffold perform specifically on targets with no direct training-set neighbor in the GO graph? The paper retains these targets in the evaluation, and this seems like the most informative regime for testing whether the learned multi-hop representation adds value beyond nearest-neighbor transfer."
Response. The learned multi-hop representation adds value beyond nearest-neighbor transfer: for genes with no direct training neighbor, scaffold predicts better than neighbor averaging (Table 3). This shows that scaffold can use indirect paths in the relationship graph, that is, information reached through other genes over several hops; neighbor averaging can use direct neighbors only.
Such genes have no neighbor in the training set, so neighbor averaging has nothing to average and its prediction is the training mean; Table 3 therefore compares with the training mean.
The official splits cannot answer this question: they contain only 20 cases with no direct training neighbor (out of 2,181), and 18 of these have no path to the training set at all (Supplementary Table S9). We therefore ran two dedicated tests, and both show that scaffold uses indirect paths (Table 3).
- Whole complexes or functional modules held out: this gives several hundred genes with no direct training neighbor in K562 and RPE1, and scaffold predicts them better than the training mean.
- Direct edges removed: with all direct edges between test genes and training genes removed on the official splits, only indirect paths remain between them, and scaffold is still better than the training mean.
| Table 3: scaffold minus neighbor averaging (here equal to the training mean), genes with no direct training neighbor | PCC-delta | MSE | Reading |
|---|---|---|---|
| Whole complexes held out (349 cases) | +0.115 [0.086, 0.145] | −0.0224 [−0.0274, −0.0174] | scaffold better on both |
| Whole GO-term gene sets held out (311 cases) | +0.056 [0.041, 0.071] | −0.0119 [−0.0169, −0.0073] | scaffold better on both |
| Official K562 and RPE1 splits, all direct test–train edges removed | +0.047 [0.040, 0.053] | −0.0048 [−0.0061, −0.0035] | scaffold better on both |
Response to Reviewer eMJR (1/2): Weakness and Question 1
We thank the reviewer for the comments. Their core concern is how to separate the value of the biological prior itself from the value added by the learned model. This corresponds exactly to the two claims of the paper, and the overall argument is given in the general response. We reply point by point below. PCC-delta and MSE are the official Top-100 DEG metrics, and brackets are 95% bootstrap intervals over targets.
Weakness. "Novelty is limited. Graph-smoothing perturbation representations followed by a linear readout is conceptually simple, and related methods such as GEARS already use GO-derived perturbation graphs."
Response. The novelty of this work does not lie in a component; it lies in posing and answering a question that is seldom asked directly. The simplicity of the method is deliberate.
One of the core tasks of perturbation prediction is to predict unseen perturbations within a given cellular context. For such a perturbation, the training data contain no response to it at all; the only thing that links it to the measured perturbations is the relationships between genes.
This relationship information can come from different sources: curated biological knowledge, such as GO; or representations learned from data, such as gene representations pretrained on very large collections of single-cell expression data. As the reviewer says among the strengths, much of the field is moving toward the latter, with ever larger models. Yet existing benchmark studies find that deep learning models do not consistently outperform simple baselines (Ahlmann-Eltze et al., 2025).
We therefore ask the opposite question: if a model does only one thing, namely use existing relationship knowledge in the simplest way, how far can it go? The answer is that a linear model with only 8,192 shared parameters attains the best mean rank among 17 methods (1.44). Using a GO graph is not by itself enough to obtain this result: GEARS also uses a GO-derived graph and has a mean rank of 12.11.
The significance of this result goes beyond the rank. scaffold is itself a simple and competitive solution. It also sets a bar: the level that can be reached from gene-relationship information alone. Any model that claims to have learned deeper mechanisms should first exceed this bar; in terms of mean rank over the nine datasets, no method currently does.
Question 1. "Does the main benchmark ranking hold when using the same study-filtered GO construction used in the relationship analyses, or using a GO release that predates each perturbation dataset?"
Response. The ranking holds under both constructions, with a mean rank of 1.44 in each.