acceptodds
Under review as a conference paper at ICLR 2027

AmbiModBench: Benchmarking Gene Perturbation Prediction Beyond Shared Responses

Abstract

Predicting cellular responses to genetic perturbations helps prioritize experiments in single-cell genomics, where exhaustive measurement is infeasible. While computational models increasingly predict these responses, three evaluation deficiencies obscure what their scores demonstrate. First, absolute metrics cannot separate target-specific predictions from a shared background response. Second, common metrics remain high under gene shuffling, so gene-level accuracy is never verified. Third, a score at one training size says nothing about coverage, which depends on representation-space proximity and response-constraining power. We propose AmbiModBench, a specificity-aware, gene-resolved and coverage-aware benchmark. It pairs every score with a training-mean reference fitted on the same split, screens each readout by gene-coordinate permutation, and links embedding distance to response variation. Across K562, RPE1 and Norman, strong absolute scores largely reflect shared background rather than target-specific learning. Widely used readouts track response magnitude distributions rather than the affected genes. Detectable gain follows representation-space coverage rather than training-set size. Nonetheless, on RPE1 the protocol yields a reproducible target-specific gain across five additional splits and three gene selections, which absolute scores alone cannot distinguish from shared background. Code and data are available at https://anonymous.4open.science/r/ambimod-4D5B/.

Primary area:
datasets and benchmarks
Keywords:
Zero-shot learning, Perturbation response prediction, Transcription factors, Model diagnosis, MSE non-monotonicity, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JtVsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

this paper proposed AmbiModBench for single-cell gene perturbation prediction benchmark to address 1) conventional absolute metrics cannot identify shared versus target-specific perturbation response, 2) some metrics are less sensitive to gene-coordinate permutation thus cannot be used to accurately measure gene-level prediciton, and 3) train split doesn't sufficiently describe the represetation space of pertubration in held-out split. thus they proposed 1) comparing model prediction results to training-mean baseline on same train split, 2) using permutation test and independent replicate calibration to check sensitivity of metrics against gene identity and measurement noise, and 3) by defining ambiguity modulus and test-to-training embedding radius to derive conditional converge bound based on Lipschitz continuity.

Critical strengths
  1. compared model predictions and training-mean baseline on same target held-out to avoid judging whether a model learnt target-dependent information by high absolute score. their rigorous design on target-disjoint train/test split and clusterwise target bootstrap provide more statistical evidence on the results.
  2. separately discussing gene aligned readouts and marginal response distributions explains why measurement noise can might turn, e.g. MSE towards collapsing to cohort mean. by calibrating on gene permutation and independent-replicate, it provdies more useful diagnostics for reference.
Critical weaknesses
  1. the paper didn't propose much non-incremental or novel methodological improvement. there has been plenty of studies proposing similarly systematic studies regarding shared perturbation response, uninformative baseline, or perturbation-specific evaluation. Training-mean baseline, psotiive or negative controls, control-relative performance comparisons and introducing confidence interval were proposed in predated literatures. and no non-trivial new conclusion are revealed by their benchmark comparing to existing results, thus it leverages the significance for its benchmark contribution effort.
  2. top 100 direction agreement seems only checked predicted signs for ground-truth top response genes, rather than requires accurate identification of genes under perturbation nor punishing FP prediction of other genes.
  3. reconducting PCA on each train split raise discrepancy on representation space under different fractions, thus the conclusion that coverage determines gain rather than sample size cannot be sufficiently supported.
Questions for authors
  1. can u provde direct empirical evidence to prove embedding coverage is boosting paired predictive performance? e.g. show distance distribution between target embedding held-out to their nearest target embedding in train split?
  2. can u compare perfromance on train subsets with different coverage while maintaining train split size the same?
  3. can u reason about the choice to compare embedding distance under different fractions that will go through separate PCA process?
  4. can u add independent replicate calibration analysis, and assess affected-gene idnetification and direction correctness?
Reviewer_SnSxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AmbiModBench, a benchmark for evaluating gene perturbation response prediction. It addresses three limitations of existing evaluations: inflated scores from shared responses, metrics insensitive to gene identity, and inadequate characterization of training coverage. The proposed framework combines training-mean baseline comparisons, gene-permutation and replicate calibration, and an embedding-based coverage analysis. Experiments on K562, RPE1, and Norman demonstrate reproducible improvements over the baseline on RPE1, no detectable improvement on K562, and inconclusive results on Norman.

Critical strengths
  1. Careful Statistical Evaluation and Reproducibility. The paper uses paired contrasts with target-cluster bootstrap intervals and validates its findings across five additional splits and three gene selections. Kernel ridge achieves a reproducible +2.31 pp gain on RPE1 (95% CI: [1.45, 3.18]), supporting the robustness of the observed improvement.

  2. Informative Training-Set Subsampling Analysis. The fixed-test subsampling experiment systematically examines how predictive gains change with training data availability. Across five RPE1 splits and five random orderings per split, kernel-ridge gain increases from +0.01 pp at 10% of training targets to +2.38 pp at 100%. This provides concrete evidence of how target-specific predictive performance evolves as additional perturbations are observed.

Critical weaknesses
  1. Limited Demonstration of Incremental Value Over Existing Evaluation Frameworks. Prior work such as Systema (Viñas Torné et al., 2026) and PerturBench (Wu et al., 2025) has already addressed shared-response biases and limitations of conventional perturbation metrics. Although AmbiModBench introduces paired contrasts, gene-permutation screening, and coverage analysis, the paper does not directly compare its protocol with these frameworks or clearly demonstrate what additional diagnostic insights it provides. Since paired subtraction preserves model rankings within the same split and readout, and the permutation screen largely verifies expected metric properties, stronger empirical evidence of their added practical value is needed to establish a substantial contribution beyond existing evaluation approaches.

  2. Limited Validation on Representative Perturbation Models. The evaluation mainly relies on ridge regression, kernel ridge regression, and a single constrained neural network, without testing established perturbation prediction models such as GEARS. Notably, the neural baseline produces residuals with only 0.95% of the ground-truth residual amplitude on RPE1 (Appendix C.1, Table 9). While the proposed evaluation principles may be model-agnostic, their practical utility and the generalizability of the empirical findings remain insufficiently demonstrated across representative model architectures.

Questions for authors
  1. Compared with Systema and PerturBench, what additional insights or model-selection decisions does AmbiModBench enable? Given that the paired score subtraction preserves model rankings within the same split and readout, could the authors provide a concrete example where the proposed protocol reveals a meaningful finding that existing evaluation approaches would miss?

  2. To what extent do the empirical conclusions extend beyond the selected regression models? In particular, given the severely limited residual amplitude of the neural baseline (Appendix C.1, Table 9), what evidence supports the practical utility of AmbiModBench for established perturbation prediction models such as GEARS? Are additional evaluations or analyses available that could clarify this?

Reviewer_hUeYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AmbiModBench combines a paired comparison with a training-target mean, gene-permutation and cell-partition calibration of metrics, and a representation-based coverage formulation. Six fixed predictors are evaluated on target-disjoint K562, RPE1 and Norman splits. Kernel ridge improves RPE1 top-100 directional agreement by 2.31 percentage points over the mean reference, with a target-cluster 95% interval of [1.45, 3.18]; additional splits and gene cutoffs retain positive gains. The paper also proves a conditional nearest-neighbor error bound and studies performance as nested subsets of training targets become available.

Critical strengths
  1. The RPE1 gain is small but supported by repeated target holdouts

The comparator is fitted from training targets in each split, and its score is paired with the model on the same held-out response. The paper distinguishes 1,308 target-split records from 792 unique targets and keeps repeated appearances together during the bootstrap. Excluding the previously evaluated seed42 still gives kernel ridge a 2.36-point top-100 gain with interval [1.49, 3.26]. The positive result also survives top-20 and top-2% gene selections. These details make the RPE1 finding more persuasive than a single split with a favorable absolute score. Norman is described as unresolved when its interval spans zero, which is the correct interpretation.

  1. The metric discussion distinguishes gene assignment from noise and amplitude

Appendix A.4 proves permutation invariance only for distances over the marginal distribution of values across genes. It explicitly excludes joint distributions of cell vectors. That distinction is necessary: a limitation of a one-dimensional response-value histogram cannot be transferred to every energy or Wasserstein evaluation. The separate cell partitions also provide a useful shrinkage example. In Table 8, squared error is smallest near mixture weight 0.2, while correlation and top-gene recall continue improving as the replicate contribution increases. Appendix A.3 retains shared-control covariance terms. The reader can therefore see both what information a score retains and why a noisy reference can change its optimum.

Critical weaknesses
  1. The coverage experiment measures a learning curve without measuring coverage

The abstract claims that detectable gain follows representation-space coverage rather than training-set size. Section 5.3 makes the same claim. But Figure 4 and Table 7 vary the retained training fraction from 10% to 100%, and report the corresponding score improvement. Neither delta_t, the nearest-training-target radius, nor the ambiguity modulus is estimated. The limitations state this explicitly. As reported, kernel-ridge gain rises from 0.01 points at 87 training targets to 2.38 points at 871 targets. That is evidence that more training responses help the fitted procedure. It does not distinguish sample count from the geometry that the paper introduces to explain it.

The argument that adding targets cannot increase the nearest-neighbor radius assumes a fixed representation. Appendix B.3 instead refits PCA and row normalization at every fraction, and also refits the model and training-mean reference. Distances across those representations need not be ordered by the set-inclusion argument. Training fraction can be an operational proxy, but the current experiment cannot attribute the gain specifically to geometric coverage or say whether equal-sized sets with different coverage behave differently.

Theorem 1 is a valid conditional result for a nearest-neighbor response estimator under a finite response-specific Lipschitz constant L. It is not an error certificate for the fitted ridge or kernel-ridge predictors. Since L is not estimated, and no empirical relationship between local embedding distance and response variation is shown, the bound cannot currently tell a user which test targets are supported. This is the central missing measurement in a benchmark named after the ambiguity modulus.

A direct test is available with the current data: fix the representation, construct equally sized training subsets with different radii, and compare gains on the same test targets. Reporting the measured radii and a training-only, noise-aware estimate of local response variation would connect the theorem to the evaluation. The coverage claim would then have empirical content beyond an ordinary learning curve.

  1. The neural result is constrained by output scale, and the benchmark's added value is not yet demonstrated

The only neural predictor is a four-layer spectrally normalized network fed unit-normalized PCA embeddings. Appendix C.1 itself explains the scale problem. Under the unit-Lipschitz bound used there, two input vectors are at most distance two apart, so their output separation is also at most two. The standardized RPE1 response residual has characteristic norm about sqrt(4063)=63.7. Such a network is constrained toward small target-dependent deviations before any conclusion about the availability of biological signal is drawn.

Table 9 matches that concern: residual amplitude ratios are 0.95% in the frozen configuration, 0.51% after removing the auxiliary penalty and 1.76% without early stopping. Both ablations preserve the spectral scale restriction. Removing the auxiliary penalty changes median residual-truth correlation from -0.021 to +0.202 while the directional score barely changes. The network can recover a response-aligned direction without a large enough correction to change signs around the mean. The current ablations therefore do not test whether a reasonably scaled neural readout can outperform the reference. The appendix calls the spectral condition soft, so the actual layer norms and learned output scale also need to be reported rather than assuming an exact bound.

An ordinary tuned MLP, or a network with a learnable output scale, would address this problem with the same inputs and splits. Applying the protocol to an established perturbation predictor would further demonstrate that it changes a substantive model-selection decision. At present the primary positive finding comes from two regression readouts, and held-out paired comparisons are limited to direction agreement. Effect-size accuracy or ranking experiments are still needed to support a claim about choosing interventions, even if a full leaderboard is outside the intended scope.

The novelty statement also needs tightening. The related work says prior studies do not compare against a reference fitted on the same split. Ahlmann-Eltze et al. already use the mean across training perturbations as a baseline. The contribution should be the particular paired reporting and its validated calibration/coverage additions. The existing baseline principle cannot carry that claim by itself.

Questions for authors
  1. With the representation fixed, do training subsets of the same size but different measured delta_t produce different held-out gains? Please report those radii, test whether they explain error after accounting for training count, and show whether nearby training embeddings actually constrain response variation without using test responses to construct the diagnostic.

  2. What happens to the neural result when the spectral restriction is relaxed or an output scale is learned? Please compare a tuned ordinary MLP with the current network and report actual layer norms, residual amplitude, residual-truth correlation and the paired directional gain. The present auxiliary-loss and early-stopping arms do not remove the identified scale constraint.

  3. Does the RPE1 gain persist after excluding targets represented by very few cells, or after propagating cell and shared-control sampling uncertainty? Please also provide one aligned effect-size or intervention-ranking comparison for ridge and kernel ridge. The current intervals correctly describe target variation conditional on fitted splits; these checks would clarify the scope of the positive scientific conclusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.