acceptodds
Under review as a conference paper at ICLR 2027

SCAFFOLD: A Simple Graph-Based Linear Model Enables Accurate Prediction of Transcriptomic Responses to Unseen Perturbations

Abstract

Predicting responses to untested gene perturbations could extend the reach of single-cell genetic screens. As recent work emphasizes large pretrained models, we investigate how existing biological relationships can guide prediction. Across two studies and four cell backgrounds, stronger Gene Ontology (GO) annotation overlap and STRING associations corresponded to more similar measured perturbation responses. We introduce scaffold, a graph-based linear predictor in which a GO annotation-overlap graph constrains gene representations while measured responses train a shared linear response map. In a unified comparison of 17 methods across nine scPerturBench datasets, scaffold ranked first on seven datasets and third on the remaining two, achieving the best mean dataset rank of 1.44 under the benchmark's six-metric ranking procedure. Controlled comparisons showed similar prediction quality with GO and STRING, while control-cell coexpression and much wider GO neighborhoods performed less well. These results demonstrate the predictive value of biological relationships within a compact linear model and identify relationship source and neighborhood selection as useful design choices alongside advances in model scale.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
single-cell perturbation prediction, transcriptomics, graph-based learning, linear models, gene perturbations, biological knowledge

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_eMJROpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

scaffold predicts expression changes for gene perturbations that were never measured. Each gene gets a small learned vector, and genes with overlapping Gene Ontology annotations are encouraged to have similar representations. A shared linear map turns that representation together with the control-cell state into a predicted response. On nine datasets from scPerturBench it ranks first on seven and third on the remaining two, ahead of substantially more complex models.

Critical strengths

I like this paper. Seeing a small linear model come out on top of a public benchmark is useful, especially when much of the field is moving toward increasingly large pretrained models. I appreciated that the authors reproduced the benchmark's published rankings before inserting their own method. The comparison of GO, STRING, coexpression, and different GO neighborhood sizes is also practically useful.

Critical weaknesses

Novelty is limited. Graph-smoothing perturbation representations followed by a linear readout is conceptually simple, and related methods such as GEARS already use GO-derived perturbation graphs. What worries me more is the neighbor-averaging baseline in the supplement. It simply averages responses from GO-connected training perturbations. scaffold has better mean PCC-delta, but neighbor averaging actually has lower mean MSE. Since this very simple comparator captures a substantial part of the signal, I think it belongs in the main comparison. It would help distinguish the value of the biological prior itself from the value added by the learned model.

The other issue is potential contamination through the GO resource. For the relationship analyses in Figures 2 and 4, the authors remove GO annotation records whose cited source is the study under analysis. The benchmark predictor, however, uses the supplied GO resource without this filtering. This is not necessarily leakage, but it creates a temporal or post-study contamination risk: if annotations were added based on results from the same perturbation screens being predicted, they could encode information downstream of the evaluation data.

Questions for authors
  1. Does the main benchmark ranking hold when using the same study-filtered GO construction used in the relationship analyses, or using a GO release that predates each perturbation dataset?

  2. Where does the GO neighbor-averaging baseline rank if it is evaluated under the complete benchmark procedure rather than reported only as a mean-only comparator?

  3. How does scaffold perform specifically on targets with no direct training-set neighbor in the GO graph? The paper retains these targets in the evaluation, and this seems like the most informative regime for testing whether the learned multi-hop representation adds value beyond nearest-neighbor transfer.

2 comments
AuthorsOpenReview ↗

Response to Reviewer eMJR (1/2): Weakness and Question 1

We thank the reviewer for the comments. Their core concern is how to separate the value of the biological prior itself from the value added by the learned model. This corresponds exactly to the two claims of the paper, and the overall argument is given in the general response. We reply point by point below. PCC-delta and MSE are the official Top-100 DEG metrics, and brackets are 95% bootstrap intervals over targets.

Weakness. "Novelty is limited. Graph-smoothing perturbation representations followed by a linear readout is conceptually simple, and related methods such as GEARS already use GO-derived perturbation graphs."

Response. The novelty of this work does not lie in a component; it lies in posing and answering a question that is seldom asked directly. The simplicity of the method is deliberate.

One of the core tasks of perturbation prediction is to predict unseen perturbations within a given cellular context. For such a perturbation, the training data contain no response to it at all; the only thing that links it to the measured perturbations is the relationships between genes.

This relationship information can come from different sources: curated biological knowledge, such as GO; or representations learned from data, such as gene representations pretrained on very large collections of single-cell expression data. As the reviewer says among the strengths, much of the field is moving toward the latter, with ever larger models. Yet existing benchmark studies find that deep learning models do not consistently outperform simple baselines (Ahlmann-Eltze et al., 2025).

We therefore ask the opposite question: if a model does only one thing, namely use existing relationship knowledge in the simplest way, how far can it go? The answer is that a linear model with only 8,192 shared parameters attains the best mean rank among 17 methods (1.44). Using a GO graph is not by itself enough to obtain this result: GEARS also uses a GO-derived graph and has a mean rank of 12.11.

The significance of this result goes beyond the rank. scaffold is itself a simple and competitive solution. It also sets a bar: the level that can be reached from gene-relationship information alone. Any model that claims to have learned deeper mechanisms should first exceed this bar; in terms of mean rank over the nine datasets, no method currently does.

Question 1. "Does the main benchmark ranking hold when using the same study-filtered GO construction used in the relationship analyses, or using a GO release that predates each perturbation dataset?"

Response. The ranking holds under both constructions, with a mean rank of 1.44 in each.

  • Study-filtered GO: only one GO annotation across the nine datasets cites a benchmark's own study (MANF, GO:0036500, Adamson). With it removed, the mean rank remains 1.44 and Adamson remains first.
  • Earlier GO release: with each graph rebuilt from a GO release about one year older than its study (2015-12 to 2021-06), the mean rank is likewise 1.44 and every dataset keeps its rank. At least 95% of target genes still have annotations, and accuracy is close: current minus earlier GO is PCC-delta −0.008 [−0.020, 0.005] and MSE +0.0004 [−0.0001, 0.0010].
AuthorsOpenReview ↗

Response to Reviewer eMJR (2/2): Questions 2 and 3

Question 2. "Where does the GO neighbor-averaging baseline rank if it is evaluated under the complete benchmark procedure rather than reported only as a mean-only comparator?"

Response. Under the complete benchmark procedure, neighbor averaging ranks second, with a mean rank of 1.89; scaffold has 1.44, and both are ahead of the 15 released methods and TxPert-public. The strength of neighbor averaging shows that GO relationships are themselves strongly predictive; scaffold improves further on top of it. These are the two parts the reviewer wishes to separate.

The 1.44 here is the rank of scaffold's predicted mean in this extended comparison; it is not the same comparison as the 1.44 in the paper's 17-method table, and the two values coincide by chance. Three of the six metrics compare cell distributions, and neighbor averaging produces no cells, so cells are generated in the same way for both methods: each gene of the control cells is shifted as a whole so that the mean of the cells equals the predicted mean. We apply the same procedure to scaffold, starting from its predicted mean, instead of taking the cells that scaffold outputs directly. Otherwise the three distribution metrics would compare two ways of generating cells, not the predictions of the two methods. We therefore report this as an extended comparison.

Over all nine datasets, scaffold's PCC-delta is 0.031 [0.016, 0.049] higher than that of neighbor averaging. Table 1 separates the two parts on K562 and RPE1, the two datasets with the most test genes: the step from the training mean to neighbor averaging is the gain from GO relationships, and the step from neighbor averaging to scaffold is the gain from the learned model.

Table 1: PCC-deltaTraining meanNeighbor averagingscaffold
K5620.4290.5340.573
RPE10.5590.5970.618

The reviewer notes that neighbor averaging has lower MSE. This holds only in the easier setting. In a harder one, where all genes of a complex or functional module are held out together, the difference reverses: scaffold has lower MSE, and its lead in PCC-delta grows as well (Table 2). The learned model shows its advantage where the task is harder. The test was run on K562 and RPE1.

Table 2: scaffold minus neighbor averagingPCC-deltaMSEReading
Random splits+0.025 [0.018, 0.033]+0.0032 [0.0022, 0.0043]PCC-delta: scaffold higher; MSE: neighbor averaging lower
Whole GO-term gene sets held out+0.082 [0.072, 0.092]−0.0030 [−0.0048, −0.0013]scaffold better on both
Whole CORUM complexes held out+0.110 [0.097, 0.122]−0.0057 [−0.0078, −0.0036]scaffold better on both

Question 3. "How does scaffold perform specifically on targets with no direct training-set neighbor in the GO graph? The paper retains these targets in the evaluation, and this seems like the most informative regime for testing whether the learned multi-hop representation adds value beyond nearest-neighbor transfer."

Response. The learned multi-hop representation adds value beyond nearest-neighbor transfer: for genes with no direct training neighbor, scaffold predicts better than neighbor averaging (Table 3). This shows that scaffold can use indirect paths in the relationship graph, that is, information reached through other genes over several hops; neighbor averaging can use direct neighbors only.

Such genes have no neighbor in the training set, so neighbor averaging has nothing to average and its prediction is the training mean; Table 3 therefore compares with the training mean.

The official splits cannot answer this question: they contain only 20 cases with no direct training neighbor (out of 2,181), and 18 of these have no path to the training set at all (Supplementary Table S9). We therefore ran two dedicated tests, and both show that scaffold uses indirect paths (Table 3).

  • Whole complexes or functional modules held out: this gives several hundred genes with no direct training neighbor in K562 and RPE1, and scaffold predicts them better than the training mean.
  • Direct edges removed: with all direct edges between test genes and training genes removed on the official splits, only indirect paths remain between them, and scaffold is still better than the training mean.
Table 3: scaffold minus neighbor averaging (here equal to the training mean), genes with no direct training neighborPCC-deltaMSEReading
Whole complexes held out (349 cases)+0.115 [0.086, 0.145]−0.0224 [−0.0274, −0.0174]scaffold better on both
Whole GO-term gene sets held out (311 cases)+0.056 [0.041, 0.071]−0.0119 [−0.0169, −0.0073]scaffold better on both
Official K562 and RPE1 splits, all direct test–train edges removed+0.047 [0.040, 0.053]−0.0048 [−0.0061, −0.0035]scaffold better on both
Reviewer_XpxGOpenReview ↗
Rating
3(+1)
Confidence
2(−2)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper propose a simple linear model that leverages gene ontology and protein-protein interaction graphs to formulate perturbation prediction problem as a graph regularised latent regression. On extensive benchmarking, the model ranks top for the task of predicting unseen perturbation effects and establishes that relationship structured information is beneficial for generalisation of linear models.

Critical strengths
  1. Effective linear baselines are very valuable for the OOD perturbation prediction task
  2. This work established the important conclusion that graph-based information is important and is useful for generalisation, which is interesting.
Critical weaknesses
  1. The method itself has very limited novelty since it's a Laplacian regularised latent space with a linear decoder with spectral initialisation. All of these components are previously established, and i think more benchmarking need to be established with other classical graph based regression model such as graph lasso or just regression over GO terms.
Questions for authors
  1. Why are the control and perturbed cells randomly paired? Can you compare to a version with pseudo-bulked mean?
  2. Generalization here is mostly interpolating over known graphs, which introduce potential leakage. What if i hold off perturbations from an entire functional modules or protein complexes?
4 comments
AuthorsOpenReview ↗

Response to Reviewer XpxG (1/3): Weakness 1

We thank the reviewer for the comments. The two concerns raised, the novelty of the method and whether generalization is merely interpolation over the graph, correspond to the two claims of the paper; the overall argument is given in the general response. We reply point by point below. PCC-delta and MSE are the benchmark's Top-100 DEG metrics, and brackets are 95% bootstrap intervals over targets.

Weakness 1. "The method itself has very limited novelty since it's a Laplacian regularised latent space with a linear decoder with spectral initialisation. All of these components are previously established, and i think more benchmarking need to be established with other classical graph based regression model such as graph lasso or just regression over GO terms."

Response. The novelty of this work does not lie in a component; it lies in posing and answering a question that is seldom asked directly.

One of the core tasks of perturbation prediction is to predict unseen perturbations within a given cellular context. For such a perturbation, the training data contain no response to it at all; the only thing that links it to the measured perturbations is the relationships between genes.

This relationship information can come from different sources: curated biological knowledge, such as GO; or representations learned from data, such as gene representations pretrained on very large collections of single-cell expression data. In recent years much of the field has moved toward the latter, with ever larger models. Yet existing benchmark studies find that deep learning models do not consistently outperform simple baselines (Ahlmann-Eltze et al., 2025).

We therefore ask the opposite question: if a model does only one thing, namely use existing relationship knowledge in the simplest way, how far can it go? The answer is that a linear model with only 8,192 shared parameters attains the best mean rank among 17 methods (Table 1).

The significance of this result goes beyond the rank. scaffold is itself a simple and competitive solution. It also sets a bar: the level that can be reached from gene-relationship information alone. Any model that claims to have learned deeper mechanisms should first exceed this bar; in terms of mean rank over the nine datasets, no method currently does. The reviewer notes among the strengths that graph-based information is important for generalization; our contribution is to establish this with a sufficiently simple model, and to make it a bar against which later work can be compared.

Table 1Model typeMean rank over nine datasets
scaffoldlinear model, 8,192 shared parameters, no pretraining1.44
TxPert-publicgraph neural network5.22
scGPTpretrained foundation model6.89
GEARSgraph neural network on a GO graph12.11

We compared both classical graph-based regression methods suggested by the reviewer, and the results agree with the two claims in the general response.

First, on the official splits and under complex hold-out, regression over GO terms comes close to scaffold but does not exceed it (Table 2). That a plain regression reaches a similar level indicates that the predictive power does come from the relationship graph itself.

Second, when whole GO modules are held out, which is a more challenging generalization setting, scaffold outperforms regression over GO terms (Table 2); under both complex hold-out and GO-module hold-out, scaffold's prediction error is lower than that of the graph lasso (Table 5 under Question 2). This indicates that, in the more challenging settings, scaffold makes more effective use of the graph than classical regression methods.

Table 2: scaffold minus regression over GO termsPCC-deltaMSEReading
Official splits, nine datasets+0.006 [−0.005, 0.016]−0.0007 [−0.0017, 0.0002]close
Whole complexes held out+0.007 [−0.001, 0.016]−0.0010 [−0.0026, 0.0007]close
Whole GO modules held out+0.031 [0.024, 0.039]−0.0014 [−0.0027, −0.0001]scaffold better on both

Regression over GO terms is a ridge regression from GO terms to mean responses; its penalty is chosen on the validation set by the same criterion used to select scaffold's checkpoint.

AuthorsOpenReview ↗

Response to Reviewer XpxG (2/3): Question 1

Question 1. "Why are the control and perturbed cells randomly paired? Can you compare to a version with pseudo-bulked mean?"

Response. Compared with a pseudo-bulk version, random pairing is slightly better: its prediction error is lower and its Common-DEGs is higher (Table 3). The pseudo-bulk version replaces each pair by the mean of the sampled pairs of that perturbation, with all other settings unchanged.

Table 3: scaffold minus pseudo-bulk versionEstimateReading
PCC-delta+0.002 [−0.005, 0.010]on par
MSE−0.0010 [−0.0015, −0.0004]scaffold lower
Common-DEGs+0.009 [0.006, 0.011]scaffold higher

For Common-DEGs, the cells of both versions are generated in the same way: the control cells are shifted as a whole so that their mean equals the predicted mean.

Random pairing is used because true pairs do not exist: a cell is destroyed when it is sequenced and cannot be measured both before and after perturbation. Pairing is only a way of drawing training samples. Each training sample consists of one control cell and one perturbed cell drawn at random, and what the model learns from them is the mean shift of each perturbation.

AuthorsOpenReview ↗

Response to Reviewer XpxG (3/3): Question 2

Question 2. "Generalization here is mostly interpolating over known graphs, which introduce potential leakage. What if i hold off perturbations from an entire functional modules or protein complexes?"

Response. scaffold does not merely interpolate over neighbors in the graph: when whole protein complexes or functional modules are held out, as the reviewer suggests, scaffold remains better than the training mean, and its advantage over neighbor averaging becomes clearly larger (Table 4). The test was run on K562 and RPE1. The other seven datasets are too small to hold out whole modules: they have only about 20 test genes per split, whereas K562 and RPE1 have 323 and 266.

Table 4: scaffold minus neighbor averaging or the training mean (as named in each row)PCC-deltaMSEReading
Random splits, against neighbor averaging+0.025 [0.018, 0.033]+0.0032 [0.0022, 0.0043]PCC-delta: scaffold higher; MSE: neighbor averaging lower
GO modules held out, against neighbor averaging+0.082 [0.072, 0.092]−0.0030 [−0.0048, −0.0013]scaffold better on both
Complexes held out, against neighbor averaging+0.110 [0.097, 0.122]−0.0057 [−0.0078, −0.0036]scaffold better on both
GO modules held out, against the training mean+0.075 [0.067, 0.083]−0.0142 [−0.0167, −0.0119]scaffold better on both
Complexes held out, against the training mean+0.087 [0.076, 0.099]−0.0199 [−0.0227, −0.0172]scaffold better on both

Neighbor averaging takes the mean of the measured responses of a gene's neighbors in the same GO relationship graph that scaffold uses, with no fitted model.

The hold-out removes every measured member of a held-out gene's module, so the model cannot draw on the measured responses of other genes in the same module. The procedure is as follows.

  • Complex hold-out: CORUM protein complexes are drawn at random and all their measured genes are placed in the test set, until the official test-set size is reached. No member of a drawn complex is kept in the training set.
  • Functional-module hold-out: the same procedure, with GO terms as the modules.
  • Control: random test sets of the same size.
  • Other settings: each design is repeated with three random seeds; the model, graph and training settings are those of the main benchmark; held-out genes keep their edges in the graph, as in the paper.

Under the same hold-outs, scaffold is also better than the graph lasso in prediction error: its MSE is lower under both hold-outs, and PCC-delta is close between the two (Table 5). The graph lasso is implemented as a network lasso on scaffold's GO graph, with its penalty chosen on the validation set.

Table 5: scaffold minus graph lassoPCC-deltaMSEReading
Complexes held out−0.000 [−0.006, 0.005]−0.0092 [−0.0108, −0.0076]MSE: scaffold lower; PCC-delta: on par
GO modules held out−0.005 [−0.010, −0.001]−0.0053 [−0.0063, −0.0043]MSE: scaffold lower; PCC-delta: close

For genes with no direct neighbor in the training set, scaffold likewise predicts better than the training mean (Table 6). Such genes cannot be obtained by interpolation from neighbors.

Table 6: scaffold minus training meanPCC-deltaMSEReading
Complexes held out, genes with no direct training neighbor (349 cases)+0.115 [0.086, 0.145]−0.0224 [−0.0274, −0.0174]scaffold better on both
GO modules held out, genes with no direct training neighbor (311 cases)+0.056 [0.041, 0.071]−0.0119 [−0.0169, −0.0073]scaffold better on both
Official K562 and RPE1 splits, all direct test–train edges removed+0.047 [0.040, 0.053]−0.0048 [−0.0061, −0.0035]scaffold better on both

The reviewer is concerned that interpolating over a known graph "introduce[s] potential leakage": a test gene's functional partners are in the training set, so the model could be reading the answer from them. The hold-outs above remove exactly this: with all partners in the same complex or module held out, scaffold remains better than the training mean and its advantage over neighbor averaging grows. A second possible source of leakage is the graph itself, if GO annotations were derived from the benchmark experiments. We checked this as well, and the ranking is unchanged: only one GO annotation across the nine datasets cites a benchmark's own study, and with it removed the mean rank remains 1.44; with the graphs rebuilt from GO releases about one year older than each study, the mean rank is likewise 1.44.

I thank the authors for a comprehensive response. I am still in the opinion that for an ICLR main track paper, which should be new method focused, the novelty of the method is insufficient. However, the authors answered all my other concerns well and the paper itself does provide important baseline/benchmarking for the perturbation prediction field. Hence I will raise my score to a 3 and but lower my confidence to 2.

Reviewer_ZvDEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SCAFFOLD uses GO-annotation overlap to connect perturbation genes, initializes their embeddings with graph SVD, and jointly fits those embeddings and a shared linear response map under a Laplacian penalty. Control-cell expression is reduced to 64 PCA coordinates. Held-out gene identities and graph edges are available during learning, while held-out response measurements are excluded; the task is explicitly transductive. The paper analyzes the relationship between prior gene similarity and response similarity, then reports results on nine scPerturBench datasets, with first place on seven and third place on two in a seventeen-method comparison.

Critical strengths
  1. The small model performs consistently across the benchmark

The shared response map has 8,192 parameters, plus 64 coordinates per perturbation, and uses no foundation-model pretraining. Its mean dataset rank of 1.44 across nine datasets is therefore useful in its own right. Section 4.2 states that all 135 released method-by-dataset ranks were reproduced before the new methods were inserted into the full comparison. Three splits and equal dataset weighting make the scope of the aggregate result clear. The result is broader than a success on one large K562 screen, and a reader can see both raw metric behavior and the final ranking.

  1. The prior-response analysis addresses shared effects and overlapping observations

Section 4.1 subtracts both the control mean and the average perturbation response before measuring pairwise response similarity. That is the appropriate check if the aim is to show that related genes share more than a generic perturbation background. The delete-one-gene jackknife also acknowledges that millions of overlapping gene pairs are not independent observations. The predictor's use of held-out nodes is plainly labeled transductive, and PCA excludes validation and test response cells. I also appreciate that the authors report a PCC-delta difference of only 0.009, with interval [-0.008, 0.027], against TxPert-public instead of portraying every metric as a clear separation.

Critical weaknesses
  1. The most relevant simple baseline is outside the headline seventeen-method comparison

The final paragraph of Section 4.2 on p. 7 says that averaging GO training-neighbor responses yields lower mean MSE than SCAFFOLD, while SCAFFOLD yields higher PCC-delta. It then explicitly states that this comparator is not in the seventeen-method, six-metric ranking. That sentence materially changes how I read the headline result: the proposed model leads the declared benchmark pool, but a simpler method using the same biological information is better on one of the main accuracy measures.

Supplementary Table S7 may quantify this trade-off, but it was not supplied. The mean-only baseline can at least be compared on PCC-delta, MSE and DEG overlap without ambiguity. If the six-metric rank is to include it, a consistent population construction must be specified for the distributional metrics, such as a declared control-population shift. Reporting the raw paired differences would be more useful than relying on the expanded rank alone. A graph-regularized regression from the same fixed graph-SVD coordinates to mean responses would also isolate the gain from jointly updating embeddings.

This comparison matters because GO-Jaccard perturbation graphs are already used by GEARS. SCAFFOLD's contribution is the effectiveness of a compact response map with this information. The correct reference is therefore a comparably simple way to transfer graph-neighbor responses. The matched prior ablations in Figure 5 are helpful, but they address which prior to use within the model; they do not settle whether this training scheme is preferable to the strongest simple graph-based predictor.

The absolute effects against TxPert-public also deserve more prominence than the rank gap. The symmetric relative MSE effect is 0.067 with interval [0.030, 0.106], while the PCC-delta interval spans zero. This supports a modest error reduction alongside similar response patterns. It is compatible with acceptance, but calls for a narrower superiority claim than the rank values alone might suggest.

  1. Independent cell pairing does not identify the cell-state map used by the predictor

Section 3.1 samples control and perturbed cells independently and pairs the lists in their returned order; those pairs are reused during training. Section 3.4 then predicts z1_hat = A*z0 + b_p, with A = I + W_z. There is no observed relationship between a particular control cell and the treated cell paired with it. The question is what information the shared A can learn under that objective.

For independent control and treated states conditional on p, expected squared error contains tr(ASigma_controlA^T) plus the treated-state variance and a mean-error term. If b_p can fit the perturbation means, minimizing the first term favors A=0: variation in the source cell makes the arbitrary paired target harder to predict. The finite embedding parameterization, graph penalty and weight decay can prevent that limit, so this is an analytical property of the objective rather than a claim that the fitted model has collapsed. It nevertheless gives a concrete reason to inspect the learned map.

The shared A gives every perturbation the same transformed control covariance before zero clipping. The model can represent different mean responses, but cannot directly learn a different covariance transformation for each intervention. In a benchmark that also scores cell distributions, that restriction needs to be distinguished from learning perturbation-specific heterogeneity. I would like to see the singular values of A, how much control variance is retained, and performance from a mean-only version or A fixed to zero. These are small changes to the present analysis and would tell us whether the source-cell input contributes beyond regularized mean prediction.

The discussion already focuses on mean responses within measured cellular contexts, which is appropriate. I would retain that scope and avoid interpreting the fitted map as a recovered cell transition. The missing supplementary controls are an inspection limit; the paper does report that graph-penalty, fixed-SVD and connectivity analyses were performed, so I am not treating those analyses as absent.

Questions for authors
  1. How large are the paired PCC-delta, MSE and DEG-overlap differences against the GO-neighbor mean comparator already mentioned in Section 4.2? Please include it in the applicable main comparisons and compare with a fixed-graph-SVD mean-response regression under identical splits and checkpoint criteria. For any distributional comparison, specify the generated population rather than silently repeating a mean vector.

  2. What does the fitted A=I+W_z do to control variation? Please report its singular values or retained-variance summary and compare with A=0 or direct perturbation-mean fitting. If the input state is useful, which reported metric and datasets show that benefit?

  3. Please make the reported connectivity controls inspectable: how do targets with no training neighbor perform, and how sensitive is the benchmark to removing GO evidence citing its source studies? Section 3.2 applies that source filter to relationship analyses but says the predictor uses the supplied resource. This is a provenance sensitivity question, not an assertion of response leakage.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZvDE (1/4): Overview

We thank the reviewer for the detailed comments. They centre on two points: whether scaffold is better than the simplest way of using the relationship graph, and what the fitted map A, which takes the control cell as input, contributes to the model. The first corresponds to the two claims of the paper, and the overall argument is given in the general response; on the second we agree with the reviewer's analysis. We reply point by point below. Results use the official Top-100 DEGs, averaged within seed and then equally over nine datasets; brackets are 95% paired bootstrap intervals over targets.

On the supplementary material. "Supplementary Table S7 may quantify this trade-off, but it was not supplied."

Response. Supplementary Tables S7–S10 are in the Supplementary Material zip, in supplementary_material/supplement.pdf and supplementary_material/tables/. Their key values are repeated below.

AuthorsOpenReview ↗

Response to Reviewer ZvDE (2/4): Question 1

Question 1. "How large are the paired PCC-delta, MSE and DEG-overlap differences against the GO-neighbor mean comparator already mentioned in Section 4.2? Please include it in the applicable main comparisons and compare with a fixed-graph-SVD mean-response regression under identical splits and checkpoint criteria. For any distributional comparison, specify the generated population rather than silently repeating a mean vector."

Response. The neighbor mean is a strong comparator, which is direct evidence for the first claim of the paper: biological relationship graphs are themselves strongly predictive. On top of this, scaffold improves further: it ranks better in the full six-metric comparison, and its advantage grows when all genes of a complex or module are held out together. The results the reviewer asks for follow in turn.

The paired differences are in Table 1: on the official splits scaffold has higher PCC-delta and DEG overlap, and the neighbor mean has lower MSE. The neighbor mean takes the mean of the measured responses of a gene's neighbors in the graph used by scaffold, with no fitted model.

Table 1: scaffold minus neighbor meanEstimateDatasets
PCC-delta+0.031 [0.016, 0.049]scaffold higher on 6/9
MSE+0.0014 [0.0005, 0.0025]neighbor mean lower on 7/9
Common-DEGs (cells generated as described below)+0.003 [0.001, 0.006]scaffold higher on 6/9, tied on 3

Included in the full six-metric comparison, scaffold has a mean rank of 1.44 and the neighbor mean 1.89, both ahead of the 15 released methods and TxPert-public. The 1.44 here is the rank of scaffold's predicted mean in this extended comparison; it is not the same comparison as the 1.44 in the paper's 17-method table, and the two values coincide by chance. Three of the six metrics compare cell distributions, and the neighbor mean produces no cells. As the reviewer requests, we specify how the cells are generated. We take the control cells of the official evaluation and shift each gene as a whole, so that the mean of the cells equals the predicted mean. We apply the same procedure to scaffold, starting from its predicted mean, instead of taking the cells that scaffold outputs directly. Otherwise the three distribution metrics would compare two ways of generating cells, not the predictions of the two methods. We therefore report this as an extended comparison.

When all genes of a complex or module are held out together, scaffold's advantage over the neighbor mean grows, and its prediction error becomes lower as well (Table 2). This shows that scaffold does not merely average over neighbors. The test was run on K562 and RPE1.

Table 2: scaffold minus neighbor meanPCC-deltaMSEReading
Random splits of the official test-set size+0.025 [0.018, 0.033]+0.0032 [0.0022, 0.0043]PCC-delta: scaffold higher; MSE: neighbor mean lower
Whole GO-term gene sets held out+0.082 [0.072, 0.092]−0.0030 [−0.0048, −0.0013]scaffold better on both
Whole CORUM complexes held out+0.110 [0.097, 0.122]−0.0057 [−0.0078, −0.0036]scaffold better on both

Training the gene representations is better than fixing them at their graph-SVD initial values: MSE is lower on all nine datasets and PCC-delta is higher on eight (Table 3). This is the fixed graph-SVD mean-response regression the reviewer asks for; the two versions use identical splits, pair counts and checkpoint rule.

Table 3: trained minus fixed representationsEstimateDatasets favoring trained
PCC-delta+0.011 [0.007, 0.016]8/9
MSE−0.0031 [−0.0039, −0.0024]9/9
Common-DEGs+0.013 [0.010, 0.016]6/9

On GEARS. "This comparison matters because GO-Jaccard perturbation graphs are already used by GEARS. SCAFFOLD's contribution is the effectiveness of a compact response map with this information."

Response. We agree with this positioning: the contribution of the paper is the compact response map, not the use of a GO graph. Using a GO graph is not by itself enough to obtain this result: GEARS also uses a GO graph, with a graph neural network, and has a mean rank of 12.11 among the 17 methods; scaffold reaches 1.44 with a shared linear map of 8,192 parameters.

On TxPert-public. "The absolute effects against TxPert-public also deserve more prominence than the rank gap. [...] This supports a modest error reduction alongside similar response patterns."

Response. We agree: relative to TxPert-public, the conclusion is a modest error reduction with similar response patterns. The symmetric relative MSE effect is 0.067 [0.030, 0.106] and the PCC-delta difference is 0.009 [−0.008, 0.027], as in Section 4.2 of the paper.

AuthorsOpenReview ↗

Response to Reviewer ZvDE (3/4): Question 2

Question 2. "What does the fitted A=I+W_z do to control variation? Please report its singular values or retained-variance summary and compare with A=0 or direct perturbation-mean fitting. If the input state is useful, which reported metric and datasets show that benefit?"

Response. We agree with the reviewer's analysis, and the measurements confirm it: A, the part of the model that reads the control cell, contributes little to mean prediction and is close to zero on the larger screens; its contribution appears in the cell-distribution metrics. This agrees with the first claim of the paper: what does the work is a compact model that relies on the relationship graph. The three results the reviewer asks for follow in turn.

On four datasets A removes more than 99% of the variance of the control cells; on the other five it retains 30–62% (Table 4). The more training steps, the closer A is to zero (Spearman correlation of −0.77 between training steps and median singular value). The values come from retraining the submitted setting, which reproduces the stored result (PCC-delta 0.5858 against 0.5857).

Table 4Largest singular valueMedian singular valueControl variance retained after A (PCA space)
K562 essential0.100.020.08%
RPE1 essential0.200.020.07%
Frangieh0.290.050.44%
Tian Inhibition0.860.060.79%
Adamson1.320.4530%
Replogle exp81.810.6240%
Papalexi1.620.7352%
Replogle exp71.630.9659%
Tian Activation1.590.8462%

Supplementary Table S1 gives a smaller value (median 9e-5) because it measures something different: S1 is variance in gene space relative to the perturbed cells, and its median is set by the two large screens.

With A removed (A=0), PCC-delta and MSE are essentially unchanged; compared with fitting perturbation means directly, without cell pairs, scaffold has slightly lower error (Table 5), and its Common-DEGs is also higher when cells are generated in the same way (+0.009 [0.006, 0.011]).

Table 5PCC-deltaMSE
Full model minus A=0+0.006 [−0.007, 0.016], on par+0.0004 [−0.00001, 0.0009], on par
scaffold minus direct mean fitting+0.002 [−0.005, 0.010], on par−0.0010 [−0.0015, −0.0004], scaffold lower

The reviewer asks whether the control-cell input contributes beyond mean prediction, and on which metric and datasets. On the mean metrics it does not: the gain in PCC-delta appears mainly on Papalexi (+0.051) and Tian Activation (+0.021). On the cell-distribution metrics it does: the full model is better than the model with A=0 in E-distance, KL divergence and Wasserstein distance on all nine datasets (Table 6). The reason is that with A=0 the model gives the same prediction for every control cell, with no cell-to-cell variation; the full model retains part of the variation among control cells.

Table 6: full model minus A=0EstimateReading
E-distance−0.97 [−1.03, −0.92]full model better on 9/9
KL divergence−18.0 [−19.1, −17.0]full model better on 9/9
Wasserstein distance−1.03 [−1.19, −0.86]full model better on 9/9
Common-DEGs+0.010 [0.007, 0.012]full model higher on 6/9

Accuracy comes from the relationship graph: removing or shuffling the graph lowers it. With the graph penalty removed, PCC-delta falls by 0.121 on K562, 0.050 on RPE1 and 0.003 on Tian Inhibition (Supplementary Table S8). With the genes on the graph shuffled, PCC-delta falls from 0.575 to 0.417 on K562 and from 0.618 to 0.560 on RPE1, that is, back to the level of the training mean (0.429 and 0.559) (Supplementary Table S10). The part above the training mean therefore depends on the relationship graph.

On scope. "I would retain that scope and avoid interpreting the fitted map as a recovered cell transition."

Response. We agree, and we keep this scope: what the model predicts is the mean response of each perturbation, which is also how the discussion section of the paper positions it. We do not interpret A as a transition of cell state.

AuthorsOpenReview ↗

Response to Reviewer ZvDE (4/4): Question 3

Question 3. "Please make the reported connectivity controls inspectable: how do targets with no training neighbor perform, and how sensitive is the benchmark to removing GO evidence citing its source studies? Section 3.2 applies that source filter to relationship analyses but says the predictor uses the supplied resource. This is a provenance sensitivity question, not an assertion of response leakage."

Response. For genes with no training neighbor, scaffold predicts better than neighbor averaging (Table 7), which shows that scaffold can use indirect paths in the relationship graph, that is, information reached through other genes over several hops; neighbor averaging can use direct neighbors only. Such genes have no neighbor in the training set, so neighbor averaging has nothing to average and its prediction is the training mean; Table 7 therefore compares with the training mean.

The official splits cannot answer this question: they contain only 20 cases with no direct training neighbor (out of 2,181), and 18 of these have no path to the training set at all (Supplementary Table S9). We therefore ran two dedicated tests, and both show that scaffold uses indirect paths (Table 7).

  • Whole complexes or functional modules held out: this gives several hundred genes with no direct training neighbor in K562 and RPE1, and scaffold predicts them better than the training mean.
  • Direct edges removed: with all direct edges between test genes and training genes removed on the official splits, only indirect paths remain between them, and scaffold is still better than the training mean.
Table 7: scaffold minus neighbor averaging (here equal to the training mean), genes with no direct training neighborPCC-deltaMSEReading
Whole complexes held out (349 cases)+0.115 [0.086, 0.145]−0.0224 [−0.0274, −0.0174]scaffold better on both
Whole GO-term gene sets held out (311 cases)+0.056 [0.041, 0.071]−0.0119 [−0.0169, −0.0073]scaffold better on both
Official K562 and RPE1 splits, all direct test–train edges removed+0.047 [0.040, 0.053]−0.0048 [−0.0061, −0.0035]scaffold better on both

The benchmark ranking is not sensitive to the provenance of GO. Applying the source filter the reviewer refers to to the predictor as well: only one GO annotation across the nine datasets cites a benchmark's own study (MANF, GO:0036500, Adamson), and with it removed Adamson remains first and the mean rank remains 1.44. In addition, with each graph rebuilt from a GO release about one year older than its study, every dataset keeps its rank and accuracy is close: current minus earlier GO is PCC-delta −0.008 [−0.020, 0.005] and MSE +0.0004 [−0.0001, 0.0010].

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank the three reviewers and the area chair for their comments. The reviews point to the same set of questions, which we address together here; details are in the point-by-point response to each reviewer.

Claim 1: biological relationship graphs are themselves strongly predictive. Taking the mean of the measured responses of a gene's neighbors in the GO relationship graph, with no fitted model, is already ahead of all 15 released methods and TxPert-public in the extended comparison, including larger pretrained models and graph neural networks. All three reviewers note that neighbor averaging is strong; this is direct evidence for this claim.

Claim 2: on top of this, scaffold is more effective than neighbor averaging. scaffold adds a compact linear model on the same graph. From different angles, the three reviewers all ask whether it really does something more: whether scaffold merely interpolates over neighbors in the graph (XpxG); whether it is better than the simplest way of using the relationship graph (ZvDE); and how to separate the value of the prior itself from the value added by the learned model (eMJR). The evidence is as follows.

  • In the same extended comparison, scaffold's predicted mean ranks first and neighbor averaging second.
  • Over all nine datasets, scaffold's PCC-delta is 0.031 [0.016, 0.049] higher than that of neighbor averaging.
  • With whole protein complexes or functional modules held out, as Reviewer XpxG suggests, which is a more challenging generalization setting, this advantage grows: from +0.025 on random splits to +0.082 with functional modules held out and +0.110 with complexes held out; scaffold's prediction error also changes from higher than that of neighbor averaging to lower.
  • For genes with no direct neighbor in the training set, neighbor averaging can only give the training mean, and scaffold predicts better.
  • Compared with the graph lasso, another classical way of using the graph, scaffold's prediction error is likewise lower under the hold-outs.

Other new results. The benchmark ranking is not sensitive to the provenance of GO: removing annotations that cite a benchmark's own study, or using GO releases about one year older than each study, leaves the mean rank unchanged.

Dear reviewers,

Thank you for your reviews. Please read the authors’ response and follow up in the discussion, briefly indicating whether your main concerns have been addressed and highlighting any remaining questions. Please also update your review and rating where appropriate.

Many thanks for your time and effort.

AuthorsOpenReview ↗

To the Reviewers and Area Chair

Thank you again for your valuable feedback and for taking the time to review our work. We wanted to kindly follow up on our responses to ensure we have addressed your main concerns. Please let us know if you have any further questions or need additional clarifications. We remain committed to improving the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.