- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper examines whether perturbation-level prediction-error rankings remain stable across biological contexts while the predictor is held fixed. It introduces a tie-aware, measurement-adjusted discrepancy, D_adj, to account for finite-cell sampling variability. Experiments on Frangieh (control, co-culture, and IFNγ), Nadig (HepG2–Jurkat), and Tian (iPSC–neuron) provide evidence of context-dependent reliability reordering. The authors also examine how cell budget affects its resolution and how reordering changes experimental shortlists.
Critical strengths
The paper offers an interesting perspective on perturbation evaluation by examining the stability of relative prediction reliability. Accounting for sampling variability is valuable, and the independent datasets strengthen the empirical evidence.
Critical weaknesses
1.The problem formulation and presentation are difficult to follow. Dense notation, repetitive qualifications, and fragmented explanations obscure the central question and contribution. The relationships among prediction-error metrics, pairwise ordering distributions, and the components of D_adj need clearer explanation.
2.The practical significance remains insufficiently established. The framework characterizes changes in model-dependent reliability rankings after target outcomes become available. However, the paper does not demonstrate how this diagnostic improves model selection or prospective decisions about reusing predictions. Although descriptive evaluation can be useful, the manuscript needs a stronger justification for why this particular quantity changes evaluation practice beyond existing measures.
3.The decision analysis does not establish incremental predictive value. The reported association between reordering and regret does not translate into improved prediction in these experiments. Moreover, the analysis contains only three unordered context-pair validation blocks, limiting the strength of generalization claims. These findings leave the proposed metric’s additional decision value unproven.
Questions for authors
Could the authors clearly explain the specific limitation in existing perturbation evaluation that motivates this work? What is the core problem this framework aims to solve, and why is it practically important? In particular, what concrete evaluation or decision-making need is addressed by measuring changes in reliability rankings? Given that the framework requires target outcomes, how should practitioners use its results, and what additional insight or actionable guidance does it provide beyond existing evaluation measures?