acceptodds
Under review as a conference paper at ICLR 2027

Finite-Measurement Transport of Perturbation Reliability Rankings

Abstract

Predictions validated in one biological context are often considered for use in another, yet perturbations with low prediction error in one context need not retain their relative accuracy after a context change. We study this ordering of perturbation-level realized error while the predictor is held fixed. Because finite cell counts can themselves induce apparent rank changes, we define a tie-aware discrepancy that compares pairwise ordering distributions and subtracts expected within-context ordering disagreement. The resulting finite-measurement estimand is zero when the two pair-state laws agree, while its unbiased estimator may be negative at finite sample size. In the Frangieh Perturb-CITE-seq dataset, reliability rankings change across control, co-culture, and IFN-gamma conditions under three complementary risk metrics. The changes exceed same-context sampling baselines, become increasingly resolvable with cell budget, and are reproduced in independent HepG2–Jurkat and iPSC–neuron raw-cell analyses. Matched source-only RBF, direct-MLP, latent-MLP, and strict source-frozen GEARS sensitivities preserve positive transport signs on their respective eligible Frangieh surfaces, although their within-metric profiles are not identical. Reliability reordering also changes finite top-k experimental shortlists. In the tested controls, however, the adjusted discrepancy did not improve held-out regret prediction beyond metric identity and mean-risk shift. A known-truth comparator audit shows why the finite-measurement correction matters: uncorrected rank statistics can declare transport under a same-context null, whereas matched-null corrections restore calibration. The framework therefore supports measurement-aware auditing of cross-context changes in model-error ordering.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
single-cell perturbation, reliability ranking, context shift, measurement uncertainty, perturbation prediction, model evaluation, rank transport, experimental prioritization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zvJ3OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper examines whether perturbation-level prediction-error rankings remain stable across biological contexts while the predictor is held fixed. It introduces a tie-aware, measurement-adjusted discrepancy, D_adj, to account for finite-cell sampling variability. Experiments on Frangieh (control, co-culture, and IFNγ), Nadig (HepG2–Jurkat), and Tian (iPSC–neuron) provide evidence of context-dependent reliability reordering. The authors also examine how cell budget affects its resolution and how reordering changes experimental shortlists.

Critical strengths

The paper offers an interesting perspective on perturbation evaluation by examining the stability of relative prediction reliability. Accounting for sampling variability is valuable, and the independent datasets strengthen the empirical evidence.

Critical weaknesses

1.The problem formulation and presentation are difficult to follow. Dense notation, repetitive qualifications, and fragmented explanations obscure the central question and contribution. The relationships among prediction-error metrics, pairwise ordering distributions, and the components of D_adj need clearer explanation.

2.The practical significance remains insufficiently established. The framework characterizes changes in model-dependent reliability rankings after target outcomes become available. However, the paper does not demonstrate how this diagnostic improves model selection or prospective decisions about reusing predictions. Although descriptive evaluation can be useful, the manuscript needs a stronger justification for why this particular quantity changes evaluation practice beyond existing measures.

3.The decision analysis does not establish incremental predictive value. The reported association between reordering and regret does not translate into improved prediction in these experiments. Moreover, the analysis contains only three unordered context-pair validation blocks, limiting the strength of generalization claims. These findings leave the proposed metric’s additional decision value unproven.

Questions for authors

Could the authors clearly explain the specific limitation in existing perturbation evaluation that motivates this work? What is the core problem this framework aims to solve, and why is it practically important? In particular, what concrete evaluation or decision-making need is addressed by measuring changes in reliability rankings? Given that the framework requires target outcomes, how should practitioners use its results, and what additional insight or actionable guidance does it provide beyond existing evaluation measures?

Reviewer_SBNqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether the ordering of per-perturbation prediction error, for a frozen predictor, survives a change of biological context (e.g., control → IFNγ in Frangieh). It defines D_adj, a tie-aware pairwise discrepancy between pair-state distributions that subtracts the within-context sampling disagreement via an order-2 U-statistic (equivalently half a squared delta-kernel MMD). Transport is detected above a same-context floor on Frangieh with a ridge/PCA predictor, replicates on Nadig HepG2–Jurkat and Tian iPSC–neuron, and holds in sign for RBF, MLP and strictly source-frozen GEARS. A calibration audit shows raw Kendall/Spearman declare transport under the null while matched-null corrections do not. Reordering changes top-k shortlists, but D_adj does not improve held-out regret prediction beyond mean-risk shift.

Critical strengths

1-The estimand is carefully defined and the calibration work is real: the authors found that their own original percentile interval was badly anti-conservative (FTR 0.55) and replaced it, and they show that raw rank statistics give near-100% false transport under a same-context null. That is a useful warning for anyone comparing rankings across contexts.

Critical weaknesses

1- The paper’s own results undercut the case for D_adj as the thing to use. Matched-null Kendall achieves FTR 0.003 and power 0.997 in the simulation (Table S12), and on the 18-row decision surface plain Kendall distance correlates better with regret (0.87) than D_adj (0.78). D_adj adds no held-out predictive value beyond mean-risk shift. So the paper establishes that reliability orderings move across contexts — which is not surprising — and that you need a null correction, but not that the proposed estimand is the right tool.

2- Presentation. I read this paper three times and still struggle to say what the primary analysis is. The main text is dominated by registration bookkeeping (“claim lock”, “ledger”, “not promoted”, “retained as provenance”), and the primary decision analysis rests on one weak linear predictor and three unordered context pairs. A short, direct paper making the Kendall-false-transport point with the Frangieh and Nadig replications would be far more useful.

Questions for authors
  • Given Table S6 and Table S12, what is the concrete advantage of D_adj over matched-null Kendall for a practitioner?

  • Source OOF fidelity for the primary predictor is ~0.24 delta-cosine. How much of the “transport” is just noise reordering among uniformly bad predictions? Would a predictor that actually works show the same effect?

  • The within-metric association between D_adj and regret is ρ = 0.26. Is the pooled 0.78 mostly metric identity?

Reviewer_Grj9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the authors study whether rankings of perturbations based on prediction error in one context are preserved in another. A shift in ranking can come from various factors such as variability from sampling and the perturbation model's ability to generalize to another context. In an effort to disentangle the two, the authors propose a discrepancy metric that compares two ordering distributions across contexts while taking sampling noise within each context into account. The authors conduct empirical experiments to demonstrate this effect.

Critical strengths
  • I believe that considering a (probabilistic) preference ranking is a valid take in this setting. Reporting raw loss/error can be less actionable/interpretable in practice.
Critical weaknesses

Clarity I found the paper incredibly difficult to read. I do not believe the quality of writing was on par with the standards at ICLR.

From the beginning, the abstract is filled with (perhaps) technical jargons and nouns that doesn't mean anything to the reader. For example:

  • "tie-aware discrepancy" (line 15)
  • "pair-state laws" (line 17)
  • "known-truth comparator audit" (line 29)

Even the after reading the contributions (lines 67-71), I did not gain any insight into what the paper was going present. Each sentence in that paragraph is extremely vague (due to the same reason as the abstract).

This continues throughout the paper and it really makes it difficult to follow the paper as a reader.

Motivation The paper introduces many quantities, but never really really motivates them (i.e., talk about their significance). For example, D{}D{}, Discrepancy (maybe?). Note that Discrepancy as a term isn't even explicitly introduced in the paper. I just inferred its name. Why should we care about discrepancy? It seems like the most important thing for the authors was to convince the reader that "formalizing reliability transport as a finite-measurement comparison of perturbation-pair ordering distributions rather than mean response fidelity" is important. The Estimand Properites paragraph seems to make an attempt at this, but is not sufficient in doing so. What can we do with the information that Dadj=0.09{}D_{{adj}} = 0.09{}?

Also, the authors them give the decomposition (5), but I don't really see the value in this decomposition. What is the significance of this decomposition?

This is a recurring theme. In fact, the authors seem to use the following sentence structure quite often:

[quantity] is [X], not [Y]

but don't really discuss why this distinction is important.

The question I am left answering is: who needs this information, how does this guide decisions, and why would that improve outcomes? Omitting this kind of discussion for an applications paper is quite critical.

Figures Figures are hard to parse. Axis lables overlap in Fig 4. And for Fig 4A, there must be a better way to represent the phenomenon, it looks quite disorganized.

Also, there aren't any real discussion of figures in Section 3? The authors only present the numerical results in the figures, but leaves the interpretation up to the readers.

Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.