acceptodds
Withdrawn from ICLR 2027

InversePerturbBench: Candidate-Aware Evaluation for Single-Cell Perturbation Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Inverse perturbation retrieval identifies the intervention behind a measured transcriptional response by ranking candidate interventions, and its success depends on separating candidates rather than on fitting responses on average. We present **InverseperturbBench**, a benchmark that measures how much intervention identity survives in a candidate library, under matched queries, matched candidate sets, and a single scoring rule. Across 41 external query actions and 1,087 candidates, measured responses recover 87.80% of interventions at rank 10, against 19.51% for an archived GEARS predictor. We call this shortfall the *identity gap*. It is not a generalization failure: on the 28 actions the predictor was fitted on, recovery is 85.7% versus 28.6%, and every predicted top-ten hit falls inside that fitted stratum. It is also not an artefact of target-gene readouts, since a target-excluded comparison retains a 16.98-point gap. Predicted first ranks concentrate on a shared translation-related response rather than on candidate-specific detail, yet removing that concentration alone does not recover accuracy. We therefore develop \cari, which rescores a frozen candidate library against the identification objective without retraining any forward model and improves genetic retrieval from 15.17% to 18.83% and drug retrieval from 24.4% to 33.3%.

Primary area:
datasets and benchmarks
Keywords:
Single cell, Perturbation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YtxvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors in this paper present a method which evaluate the inversed retrieval performances of perturbation prediction data.

Critical strengths

The experiment is easy to follow, and the figure is clear.

Critical weaknesses

The authors in this paper present a method which evaluate the inversed retrieval performances of perturbation prediction data, while the idea is interesting, I’m doubting its practical usage and the method seems very naïve.

  1. The authors should clarify the main contributions and application scenarios of this design. In most cases, the inverse prediction is not very helpful given the expectation of perturbation modelling. Moreover, the causal relationship is also not clear. If there is an unknown perturbation source, we know nothing about the real perturbation effect.
  2. The evaluated data are quite limited. The authors should consider large-scale datasets such as Tahoe and Xatlas.
  3. The name of the dataset is also incorrect. GEARS is a method, and it utilizes perturbation datasets from public resources.
  4. I also cannot find updated information on selected methods; e.g., the authors should also consider methods such as STATE for generating matching data.
  5. It will be better if we have more insights which can help us design better perturbation prediction or retrieval models.
Questions for authors

Please see my questions above, I am doubting the design of this question.

Reviewer_nea5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats perturbation prediction as retrieval. A measured response is ranked against alibrary of candidate responses scored by top-k recovery of the true intervention. The "identity gap" is the difference in recovery between a measured and a predicted library with queries, candidates and scorer fixed. On 41 Adamson queries and >1K Replogle candidates measured responses reach 87.8% Exact@10 vs 19.5% for an archived GEARS export. CARI then rescores a frozen library, fusion with a listwise query encoder on Replogle K562 and a popularity calibraton on Srivatsan.

Critical strengths
  • I like the measured-library reference. Without a ceiling I never know whether 20% Exact@10 is bad or just what the data allows. Swapping only the origin of the candidates (Sec 3.1, Fig. 3) answers that directly.
  • The readout analysis is careful. 41 target transcripts alone give 75.7% Exact@10, and removing them drops recovery from 58.3% to 40.9% vs 59.1% for a matched control panel. Centering and fusion at k=1 go against them and are reported anyway.
Critical weaknesses
  • The claim I have most trouble with is that the gap "is not a generalization failure". It rests on one archived GEARS export and 28 fitted actions. As I read it the measured library comes from the cells GEARS was trained on, so the fitted-stratum gap says this run doesn't reproduce its training responses. The authors read this as lost a "candidate-specific detail" L309, but plain underfitting looks the same, and Sec 4 reports no training fit, seeds nor second model. So "a property of the predicted responses" (Sec 7) could just be a property of this checkpoint, or of GEARS. Fig. 4 also shows generalization matters bc median predicted rank is 32.5 on fitted and 426 on held-out actions. AIUI that comparison keeps the target transcripts (Sec 6.1 names only the other two as trans-only). "Not an artefact of target-gene readouts" rests on a different predictor (3-NN "semantic") and candidate set (2,021), where the gap is a net 9 of 53 queries.
  • CARI isn't shown to close the gap the paper defines. Neither variant runs on an instance with a measured reference (Fig. 6 top), so "closes part of the gap" has no real number behind it? CARI-cal isn't compared to a standard hubness correction such as Conneau et al., 2018 (cited in Sec 2). The fusion gain is 3.66 points over the authors' own 3-NN library, and Sec 5.1 has no encoder-only (lambda = 1) row under its trans-only protocol so I can't really tell which part does the work.

The evaluation idea is good. For a benchmark I'd want more than one learned model per instance, a trans-only number for the headline, and CARI measured on the gap itself.

Questions for authors
  1. (experiment) What are the measured and GEARS Exact@10 levels on the 41 queries under the global trans-only readout? If a large gap survives here and the fit in Q2 looks fine, I'd probably raise my score.
  2. (experiment) How well does the GEARS export fit its Replogle training responses, and does the fitted-stratum result hold across seeds? A well-fit model still near 29% would convince me.
  3. (clarification) Why does the measured library drop from 87.8% (cosine, 1,087 candidates) to 39.6% (Spearman, 2,021)? How was each scorer chosen?
Reviewer_hj2KOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a benchmarking method to evaluate inverse perturbation design, where a set of query effect and candidate effects are given. Among the candidate perturbations, an algorithm should ideally find the one(s) corresponding to the query effect. The paper revolves around a phenonem referred to as "identity gap" in the paper. Afterwards, the paper proposes a method called CARI that alleviates the identity gap by weighting the candidates.

Critical strengths
  • The paper proposes a standardised workflow to evaluate inverse perturbation retrieval.
Critical weaknesses
  • I failed to understand the phenomenon called "identity gap". For example looking at Fig. 4a, the difference of Exact@10 numbers between measured observations and that of the forward model is clearly attributed to the held-out actions, so that clearly points to generalisation issue in the forward model. Any comment on that?

  • In sum, I failed to follow the paper after careful reading and a lot of effort. If the method and rational is expalined during the rebuttal and are valid, I'm open to increasing my score.

Questions for authors

Please see above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.