- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper reframes unseen-perturbation response prediction as an information-decoding problem and asks whether prediction performance is primarily limited by model capacity or by the perturbation-specific information available to the predictor. Experiments across CRISPR and chemical perturbation screens show that increasing decoder capacity provides limited gains for the tested models, whereas a small number of measured genes from the new perturbation can substantially reduce prediction error. The paper further proposes a measurement metric for quantifying the value of prior information and shows that measurement performance can be predicted reasonably well from training data when the training set is sufficiently large.
Critical strengths
-
Recasting unseen-perturbation prediction as a problem of available information rather than solely model capacity provides a useful perspective and connects prediction with experimental design.
-
The proposed measurement equivalent offers an intuitive way to compare heterogeneous priors with direct experimental measurements. The finding that a few carefully selected genes can outperform the tested zero-shot priors is particularly relevant for experimental design.
-
The paper develops theoretical results characterizing the zero-information limit, linear measurement channels, finite-sample prediction error, and a coverage floor for predictors that combine training responses.
Critical weaknesses
-
The experiments consider several linear/kernel methods, MLPs, frozen pretrained representations, and a reduced-budget GEARS experiment, but do not evaluate fine-tuned foundation models or several competitive perturbation models. Therefore, the evidence more directly supports the claim that additional capacity provides limited gains among the tested models and representations, rather than a broader conclusion about model capacity in general.
-
Gene panels are constructed from whole-transcriptome pseudobulk measurements rather than validated using an actual targeted measurement platform. Consequently, the reported advantage of a few measured genes may depend on measurement noise and assay characteristics that are not fully represented in the current experiments.
-
The training-only prediction becomes noticeably optimistic with small training sets and can fail substantially under pathway-level distribution shifts. This limits its reliability precisely in settings where experimental guidance may be most valuable.
Questions for authors
-
How would the main conclusions change with fully trained competitive perturbation models or fine-tuned foundation models? In particular, could stronger models extract more useful information from the same perturbation descriptors?
-
How robust is the reported measurement advantage under realistic targeted-assay noise, dropout, and measurement constraints?
-
Is it possible to construct a training-only diagnostic for coverage failure? Currently, the coverage floor becomes observable only after responses from the novel class have been measured, which limits its use for prospective experimental design.
-
Given that the normalized error is energy-weighted and some chemical screens contain a small number of strong responders, would the main conclusions remain unchanged under an equally weighted per-perturbation metric?
2 comments
I appreciate the authors' responses and the additional clarification they provided, I will keep my positive rating for your manuscript. By the way, I am interested in whether the used perturbations are common in this field, I would be sincerely grateful if you can answer (note that it is not necessary to respond).
Three new analyses for Q2 to Q4 (simulated assay dropout and noise, a training-only coverage warning, an equal-weight metric); the capacity claim stated against the tested models and published fine-tuned benchmarks
We thank the reviewer for a precise review. Each question led to a change in the revised PDF (new text in blue): three new analyses (Tables 35 to 37; code in the supplement), explicit scope statements and, for readability, a setup sketch (Figure 1).
Q4: Equal weighting (new, Table 35). Each perturbation counts once: its error relative to that of the mean response, rp, is averaged over perturbations.
On this scale, which has no noise correction and so pulls every error toward 1, the margins are smaller but keep their sign, with intervals that exclude zero on every screen. The priors' gains sit mostly in strong responders: per perturbation no tested prior beats the mean response on A549 and MCF7, and on the chemical screens four genes gain most in the weakest third. The exception is the strongest third of Replogle, where Gene Ontology is level with or slightly ahead of four genes (+0.02 [0.00, 0.04]).
Q2: Assay noise and dropout (new, Table 36). We read the four genes from cells and corrupt every cell's readout, for training and held-out perturbations alike (dropout to zero expression; noise that multiplies the cell variance by 4), refit the decoder, and compare with the best prior on the same perturbations (paired bootstrap; (w) worse than the prior, (ns) not significant, unmarked better):
Read from up to 128 cells, one corruption costs 0.04 to 0.11 and keeps the advantage on four screens; heavy corruption of a 10-cell readout erases it. The advantage thus depends on depth and readout quality, as you anticipated, and Replogle is the fragile case (it also loses from 5 cells, Table 18). This is a simulation on real cells, not a targeted platform, whose own biases and normalization we did not test (Limitations).
Q3: Training-only coverage diagnostic (new, Table 37). Before measuring, we compare the mean fingerprint similarity of a batch of new compounds to the training compounds with random batches drawn from training only. The alarm fires on 28 of 48 pathway hold-outs and 0 of 15 random folds, but the hold-outs it misses are still optimistic by 0.33 to 0.46. No rule that sees only descriptors can certify coverage in general: two classes with the same descriptors but different responses look alike before they are measured. A missing annotation (no training member of the pathway or target class) is the conservative trigger; here it is exact by construction. For such a class, the first members should be profiled in full; with a quarter of a pathway in training the forecast's optimism fell to 0.02 to 0.13 (Table 26).
Q1/W1: Stronger models. The submission already restricted the capacity claim to the tested families; the revision makes the gaps explicit (Section 5.1, Limitations). Tested: MLPs up to 7M parameters (at most 0.010 better, Table 30), frozen ESM-2, Geneformer and ChemBERTa (at most 0.7 genes, Table 28), the language-model-based GenePT prior (0.97 on the Norman singles, Table 29) and GEARS at a reduced budget. We did not fine-tune foundation models; published benchmarks, including fine-tuned scGPT and scFoundation, report the same pattern (Ahlmann-Eltze et al., 2025; Csendes et al., 2025). Whether a stronger model extracts more from the same descriptors is open, and Theorem 1 does not forbid it; the measurement equivalent prices any such model in genes on our folds.
W3: Small training sets and pathway shift. Section 6 now states the rule: optimism 0.01 with at least 200 training perturbations (only Replogle is that large), 0.07 with 100 to 199, 0.11 below 100 (Table 7). At smaller n, the closed-form leave-one-out estimate, also training-only, has a bias of only -0.05 to +0.03 on every screen, including n=81 (Table 9). Under pathway hold-out all three degrade: four genes rise to 0.68 to 0.93 (0.25 to 0.40 under random folds), still below the priors (1.02 to 1.04, Table 25), and the forecast turns optimistic by 0.35 to 0.57, which the warning flags in part.
We would welcome further questions.