acceptodds
Under review as a conference paper at ICLR 2027

ShrinkageLens: Information Sources in Perturbation Discrimination Scores

Abstract

Perturbation discrimination ranks held-out responses against a prediction, often after magnitude correction. A corrected score can, however, obtain perturbation-specific information from scoring inputs instead of the prediction. ShrinkageLens separates these sources by specifying each rule's inputs and comparing scores using identical predictions, matched permutations and the inverse of a known shrinkage map. Identical predictions score exactly among candidates under official ties for nine rules without row-specific inputs. Scaling each prediction against its own truth raises the shared response from near 0.51 to 0.80-0.84, whereas own-truth gene weights lower it. On five Perturb-seq panels, magnitude-stratified permutations raise most null means but move no own-truth score across its range. Of 78 published model-context pairs, 32 beat the mean and 28 of these exceed their global matched null (25 under strata). GEARS on one RPE1 split and scBERT, whose predictions are effectively constant, do not. Inverting an unclipped empirical-Bayes map on positive-weight coordinates restores the unshrunk score within in all 90 settings. Scalar recovery varies with panel, center and weights. These controls locate a score's information source without ranking corrections by biological validity.

Primary area:
datasets and benchmarks
Keywords:
single-cell perturbation prediction, perturbation discrimination score, evaluation metrics, empirical Bayes shrinkage, permutation null, virtual cell

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uZxcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines perturbation discrimination scores (PDS) using identity-free predictions, matched permutation nulls, and inversion of a known shrinkage map. It asks whether a corrected PDS reflects model predictions or information introduced during scoring.

Critical strengths

1.The identity-free control and matched nulls provide a clear way to test whether a high PDS contains perturbation-specific directional information.

2.The methodology and reporting are careful. The paper distinguishes support-derived from truth-derived inputs and provides a label-free inverse control for a known empirical-Bayes map.

Critical weaknesses

1.The practical implication of the shrinkage analysis is unclear. A high raw PDS still indicates that the final output distinguishes the correct perturbation. Inverse recovery shows that an internal estimate is recoverable, but not why the lower PDS of the deployed shrunk output should change its interpretation.

2.The empirical scope is narrow: five Perturb-seq panels and an external archive reusing three screens. Moreover, the inverse control requires an explicit empirical-Bayes map, so its relevance to typical black-box predictors is unclear.

Questions for authors

1.On a common candidate library and gene space, how often do raw PDS, normalized PDS, and matched-null conclusions change external model rankings?

2.When a normalization uses the true response of the evaluated perturbation, should the resulting score be removed, labeled as a conditional metric, or reported alongside raw PDS?

3.For black-box predictors without an explicit invertible map, which components of the framework remain applicable, and do they change conclusions beyond raw PDS and response-error metrics?

7 comments
AuthorsOpenReview ↗

Author Response to Reviewer uZxc

Thank you for these questions. On common candidate libraries, normalization changes point rankings, but conditional intervals resolve few reversals. Scoring-input controls apply to black-box predictors without an inverse. None of this invalidates the raw PDS of a deployed shrunk output. Inverting a known map tests precursor recoverability and does not imply that its score should replace the deployed output's score.

1. Common-library rankings and null conclusions. In new Appendix Z.2 (Table Z.2), we reanalysed the original GEARS, scGPT, self-trained linear and mean predictions across six contexts sharing one candidate library and 1,000 genes per context (Adamson 21/21, K562 107/108, RPE1 171/164), reproducing all 16,576 retained per-perturbation values exactly across the three reused screens.

Each transition below spans 36 model-pair/context comparisons.

TransitionL1 point reversalsL2 point reversalsL1 opposite-direction unadjusted 95% intervals
Raw → training-fixed7/36 (three contexts)5/360
Raw → own-truth9/36 (all six contexts)6/361

Intervals are exploratory and conditional on fixed predictions and libraries, so point flips need not be resolved reversals. Direct paired L1 gap changes exclude zero in 11/36 raw-to-training-fixed and 18/36 raw-to-own-truth comparisons, measuring gap shifts rather than resolved flips. For scGPT minus mean on common-library RPE1 split 1 (N=171), the gap moves from +0.04808 [0.02551, 0.07257] under raw L1 to −0.01594 [−0.03201, −0.00034] under own-truth L1, a paired change of −0.06402 [−0.08577, −0.04210], showing scoring sensitivity rather than which ordering is biologically right.

Matched-null status on common libraries is separate (Tables L.3, L.5, L.7). Raw-to-training-fixed L1 changes 1/18 non-mean statuses, GEARS on 164-target RPE1 split 2 from inside (0.5625 in [0.5610, 0.5701]) to below (0.4955 vs [0.4990, 0.5071]). Raw-to-own-truth L1 changes 1/18, scGPT on Adamson split 1 (N=21) from inside (0.5692 in [0.4807, 0.5714]) to above (0.6349 vs [0.4853, 0.6009]). We do not rank by null subtraction, and the 78-pair method-specific libraries cannot replace common-library rankings.

2. Reporting truth-dependent normalization. We recommend keeping it only alongside raw PDS, response error and effect recovery, labeled a truth-conditioned diagnostic with its identity-free control and matched direction-null. Because it reads the receiving perturbation's held-out norm before forming distances, its discrimination cannot be credited solely to deployment-available predictions.

3. Black-box use and the resulting decision. Input declaration, identity-free controls, matched global and stratified permutations, and effect-recovery reporting need only predictions and aligned candidate responses, not a known map. Inversion is omitted when the map is unknown, since our inverse applies only to the explicit empirical-Bayes map and the noisy training-pair proxy fit passed in 23/30 settings, failing its all-settings criterion (Appendix P.1).

These controls change conclusions beyond raw PDS and response error alone. On the full 384-target RPE1 split 2, GEARS beats the mean on raw L1 by 0.0652 [0.0371, 0.0944] (0.5665) yet lies below its global direction-null [0.5666, 0.5723] and inside its stratified null (Table M.3), and beats the mean on own-truth L1 by 0.0223 [0.0145, 0.0302] (0.7856) inside [0.7818, 0.7872] while losing 0.043 on top-20 effect overlap (Appendix Z.3), whereas scBERT's effectively constant predictions beat the mean on K562 only via own-truth norms (Appendix M). We retain the reported gains while withholding directional attribution, without invalidating raw PDS or claiming biological signal is absent.

Numerical correction and stability. Repeating the 78-pair family under 20 new 2,000-permutation streams leaves 464/468 rule-by-reference cells and every stratified own-truth class unchanged (Appendix Z.1). Separately, on Adamson split 2 (N=13), the RPE1-embedding linear model has an exact zero own-truth bootstrap lower bound on the PDS rank lattice rather than 8.54e-18, lowering own-truth mean-beaters from 32 to 31 (27 above global null, 25 above stratified, while raw and training-fixed stay 31 and 32, Table M.1). Evidence remains scoped to the five internal panels and three reused external screens (including post-hoc top-20 effect recovery in Appendix Z.3, where within-pair score-overlap medians are 0.59 and 0.50 for training-fixed and own-truth versus 0.22 for raw L1).

Thank you for the detailed rebuttal and additional analyses, which have helped clarify several of my earlier concerns.

The common-library experiments demonstrate that different scoring rules can affect model rankings, and these changes primarily reflect scoring sensitivity. I also appreciate the clarifications regarding truth-dependent normalization and the framework's applicability to black-box predictors.

Nevertheless, I remain concerned about the incremental value and practical significance of the proposed diagnostic framework. While the analyses identify potential limitations in interpreting raw PDS, it remains unclear how shrinkage works well in practice. Overall, although the rebuttal has addressed some of my concerns, I have decided to maintain my original score and confidence level.

AuthorsOpenReview ↗

Shrinkage accuracy and the evaluation decision

Thank you for pinpointing what remained unclear. In our experiments, shrinkage works well in practice by substantially reducing prediction error on noisy Perturb-seq profiles, and that accuracy gain is precisely what creates the retrieval scoring dilemma our framework addresses:

  1. How shrinkage works well in practice (Appendix E, p. 21; Table E.1, p. 22):

    • Consistent error reduction: Empirical-Bayes shrinkage achieves lower squared error and MAE than the unshrunk kk-shot mean in all 30 dataset-by-support-size settings, cutting squared error by 5.7× to 12.5× at one support cell. Even at k=32k=32 (Table E.1), it beats the unshrunk mean on Pearson correlation, MAE, and DE sign accuracy across all 5 panels (e.g., raising DE sign accuracy from 0.540 to 0.856 on Norman and from 0.593 to 0.792 on Arc-H1).
    • Opposite raw PDS outcome: Despite losing on accuracy, the noisy unshrunk mean ranks 1st in all 10/10 raw PDS comparisons (five panels across L1 and L2), while every shrinking, projecting, or trained estimator wins 0/10. As you noted, raw PDS still accurately describes retrieval by the deployed output; it simply penalizes contraction toward the shared cellular response that improves prediction accuracy.
  2. Practical decisions changed by our black-box controls (Sections 4–5, pp. 5–8, no inverse or retraining needed): Because error-reducing predictors lose raw PDS, recent benchmarks turn to post-hoc normalizations or mean-baseline comparisons. Our controls prevent two concrete benchmark misinterpretations:

    • Catching test-norm leakage in post-hoc "fixes": Matching each prediction to its evaluated target's true magnitude lifts a zero-information constant prediction by up to +0.165 above chance (Table 1), allowing effectively constant scBERT archive outputs to beat the mean on both K562 splits (Section 5, p. 7).
    • Separating magnitude alignment from gene-specific direction on raw PDS: Even on unmodified raw L1 PDS, GEARS beats the mean by +0.0652 on 384-target RPE1 split 2, yet fails to exceed either its global or magnitude-stratified direction-null (Appendix M) while scoring 0.043 below the same-library mean on top-20 effect overlap (Appendix Z.3).
  • Practical takeaway: Keep raw PDS alongside prediction error and effect recovery, label own-truth normalization as a truth-conditioned diagnostic, and require direction-null clearance before crediting a mean-beating gain to learned perturbation identity.
Reviewer_YQy5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The perturbation discrimination score (PDS), introduced by Arc’s 2025 Virtual Cell Challenge, checks whether a prediction for perturbation i is closer to the true response of i than to the true responses of the other perturbations. People have noticed that the PDS is sensitive to the size (norm) of a prediction. Various frameworks have emerged to “fix” it, for example by rescaling every prediction to the norm of its own true response (Liu et al.). This paper analyses whether the corrected perturbation discrimination score (PDS) obtains perturbation-specific information from model predictions or uses any additional inputs provided to the model during scoring. The authors introduce three “checks” for this:

  1. Give every perturbation the same prediction.
  2. Shuffle predictions across perturbations (keeping norms fixed) and rescore.
  3. Empirical Bayes shrinkage case study.
Critical strengths
  1. Timely and important problem. How to evaluate perturbation response models is a pressing question in a very active field. Commonly used metrics have been shown to correlate poorly with true biological response recovery (or at least with recovery of true differentially expressed genes (DEGs). Indeed, Dibaeinia et al. (2026) report a correlation of 0.38 between Pearson δ\delta and DEG F1, and -0.13 between L2-Δ\Delta and DEG F1. Therefore, work that analyses evaluation metrics could have a high impact.

  2. The authors demonstrate that corrected PDS can reflect information supplied during evaluation rather than variation in model predictions. The identity-free control shows that own-truth norm matching can produce above-chance discrimination for constant predictions, while own-response gene weighting also changes the reference level, in the opposite direction in the reported experiments. For example, the constant mean baseline reaches approximately 0.76 on RPE1 under own-truth normalisation. This is a useful warning against interpreting corrected PDS as direct evidence of perturbation-specific information in model outputs.

Critical weaknesses
  1. Clarity and structure. The paper is very difficult to follow. Although the abstract and introduction identify the central concern - that corrected PDS can obtain perturbation-specific information from scoring inputs - the subsequent presentation obscures this useful message beneath numerous scoring variants, bounds and recovery analyses. Terms such as “gauge group”, “bundle null” and “tie-optimistic” are quite unintuitive, and essential details of the empirical-Bayes estimator are deferred to the appendix. The relationship between the identity-free controls, matched permutations and known-map recovery is not explained clearly enough. Consequently, substantial effort is required to understand what each analysis establishes and how it contributes to the practical evaluation problem.

  2. Limited practical and biological implications. The paper ends with reporting recommendations, but their effect on real conclusions is shown only indirectly. For example, the per-prediction bound changes the top estimator in 11 of 30 settings, but the paper cannot say which ranking is correct. More fundamentally, the paper never connects its findings to the question that matters most for this field: whether a score reflects a model’s ability to recover meaningful biological responses. Without that link, it is unclear how researchers should act on the results.

Additional concerns

  • Interpretation of archived predictions. The published-model analysis relies on externally generated prediction arrays. Please identify the exact archive release and source configurations, including checkpoints where available. In particular, the effectively constant scBERT outputs should be interpreted as a property of the evaluated pipeline and prediction archive; without further checks, they should not be generalised to the model itself.
  1. Limited technical contribution and generality. The scale sensitivity of PDS is already established (Liu et al., 2025), and the authors state that they add controls, “not a new account of scale”. Moreover, the known-map inverse applies only to the authors’ own EB estimator. It does not carry over to published models, whose map is unknown. Learning the map from training pairs fails in 7 of 30 settings. The scalar-recovery ordering adds an empirical observation, but rests on five panels and is only partially stable under resampling. Overall, the audit provides useful observations, but I am not convinced that their incremental significance and generality, particularly for the map-recovery component, are sufficient for acceptance.
Questions for authors
  1. Can you discuss what your findings imply about whether PDS, corrected or not, reflects a model’s ability to recover biologically meaningful responses (e.g., true DEGs)?
  2. What is the intended practical use of the known-map inverse? For a model whose map is unknown, what would one do with it? Is there a realistic setting beyond your own EB estimator where this control applies? More broadly, is ShrinkageLens meant only as an audit of scoring inputs, with the inverse as a positive control, or is it meant to be used more widely?
  3. The GEARS RPE1 example changes the interpretation of a gain over the mean baseline. What evaluation decision should follow from this result? Please provide a worked example showing what one would report or conclude differently, and explain what remains unresolved about biological prediction quality.
  4. Please summarise the provenance and training setup of each published method (or point to the exact source configuration), and state the archive version or DOI used.

Suggestions:

  • Restructure the paper around one clearly defined main claim, with the remaining analyses in a supporting role.
  • Simplify the language. Defining all terms in the main text and stating each section’s takeaway up front would make the paper a lot easier to read.
  • Add a short checklist of practical steps for evaluating perturbation models with these controls, ideally with a worked example on a public benchmark.
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer YQy5

The GEARS case changes what can be attributed to the prediction. On 384-target RPE1 split 2, own-truth L1 beats the same-library mean by 0.0223 [0.0145, 0.0302], yet its 0.7856 score lies inside the matched direction-null [0.7818, 0.7872] (mean 0.7846, 95% of the gain). Raw L1 gains 0.0652 at 0.5665 (below its global null [0.5666, 0.5723], inside stratified), training-fixed L1 loses 0.0067, and top-20 overlap loses 0.043. We report the gains alongside raw PDS, response error and effect recovery while withholding directional attribution, leaving cell-level DEGs and mechanism unresolved.

1. Biological meaning. PDS measures retrieval within a candidate library. It is not validated as a measure of true DEG recovery. DE metrics already disagree with PDS (Table E.1), and per-prediction L2 bounds change the top estimator in 11/30 settings without identifying biological truth (Appendix J). The post-hoc 78-pair analysis (Appendix Z.3) uses the share of the truth's top 20 absolute effects among the prediction's top 20 on 1,000 genes. Gains over the same-library mean correlate at 0.49 [0.25, 0.68] for raw L1, 0.57 [0.34, 0.75] for training-fixed and 0.55 [0.31, 0.73] for own-truth. Intervals resample pairs within fixed contexts, condition on three screens and ignore related-method dependence. Both endpoints use the same measured truth. Of raw-L1 mean-beaters, 26/31 have a positive top-20 gain, as do 21/47 others. GEARS also loses 0.057 on top-50 overlap and gains 0.015 on top-20 Pearson. These are pseudobulk rankings, not cell-level DE tests. Across-pair correlation intervals overlap. Within the same 78 pairs, per-perturbation score-versus-top-20 correlations have medians 0.59 training-fixed, 0.50 own-truth and 0.22 raw, without intervals. This supports reporting effect recovery beside PDS, not independent biological validation of a rule.

2. One main claim and a checklist. Taking magnitude sensitivity from Liu et al. as given, we add an exact official floor for identical predictions under row-independent scoring (Proposition 2) and matched permutations separating mean-beating gains from directional identity. Neither needs an invertible model. The revision centers on a rule's source of perturbation-specific information, puts the controls before the supporting case study in Section 6 and defines terms at first use. Tie-optimistic scoring credits ties to bound official PDS, bundle nulls swap whole predictions and direction nulls swap directions at the receiving norm. Section 5 gives four steps.

  1. Declare the inputs each rule reads.
  2. Run the identity-free control, testing whether identical predictions gain discrimination from row-specific inputs.
  3. Run matched permutations, testing directional information beyond retained magnitudes.
  4. Report raw PDS, response error and effect recovery beside any corrected score.

All four apply to black-box models without an inverse.

3. Scope of the inverse. The inverse is a label-free positive control for the known, unclipped empirical-Bayes map on positive-weight coordinates (Section 6). It neither replaces the deployed output's raw score nor inverts unknown maps. The training-pair fit regressed outputs on a noisy response proxy rather than the map's own input. It passed in 23/30 settings and failed its all-settings criterion (Appendix P.1). This failure concerns that proxy fit, not all ways to identify unknown maps. The inverse and scalar ordering are supporting analyses.

4. New checks and a correction. Appendix Z.2 adds common-library rankings (7/36 and 9/36 L1 point flips, 0 and 1 with opposite-direction intervals), and Appendix Z.1 tests 20 permutation streams (464/468 cells and all stratified own-truth classes unchanged). An exact-lattice audit moves one Adamson split 2 lower bound from 8.54e-18 to zero, lowering own-truth mean-beaters from 32 to 31 (raw and training-fixed stay 31 and 32).

5. Archive provenance and scBERT. We used Zenodo v5, published 18 July 2025, DOI 10.5281/zenodo.16092690, specifically single_perturbation_results_parameters.RDS and single_perturbation_results_predictions.RDS, with method setups in the publication-version benchmark launcher for all 13 scored methods (CPA, GEARS, Geneformer, scBERT, scGPT, 4/33-layer UCE and six linear models).

We keep source train/validation/test splits 1/2 on Adamson, K562 and RPE1 without retraining (Appendix L and Tables L.2 and M.2 detail alignment, the 1,000 genes and exclusions). Not every source checkpoint was independently verified, and the effectively constant scBERT outputs characterize this archive and pipeline, not scBERT generally.

The rebuttal addresses each of my points directly, which I appreciate and thank the authors for. From the rebuttal, I conclude that the metrics defined in the paper do not serve as a measure of biological recovery, and that the known-map inverse is not a general tool that can be applied to any published model. Given the inverse and the scalar-recovery ordering are demoted to supporting analyses, the remaining contributions are the exact chance floor for identical predictions, the matched permutation nulls, and the two cautionary examples, GEARS and scBERT. As it stands, I do not believe these are sufficient contributions for a main track paper at ICLR. In my initial review, I recommended focusing on clarity and structure. I maintain this point, because the rebuttal itself is hard to understand. I think this paper could do well if it were rewritten with a clearer narrative, where the reader is clearly told what question is being asked, why each experiment is there, and what to take away from it - rather than being left to assemble the argument from the results themselves. The rebuttal describes restructuring along these lines, but this goes beyond a revision I can assess over a rebuttal period, so I will not be recommending acceptance in the paper's current form.

AuthorsOpenReview ↗

The evaluation question and practical contribution

Thank you for your candid feedback. You are right that our initial reply was too dense and left the reader to piece the story together from caveats. Here is the single question the paper asks, why each experiment is there, and what to take away from it:

  • Core question: Perturbation benchmarks increasingly rank foundation and causal models by retrieval (whether a predicted profile is closer to the true perturbation's response than to other candidates), often after rescaling predictions. Did a high retrieval score come from the model's predictions, or from test-set information supplied during scoring?
  1. Why shrinkage creates the benchmark dilemma (Section 6 & Appendix E): Perturbed cells share a strong common response. Shrinking noisy estimates toward that shared response lowers squared error and MAE in all 30 tested settings (by 5.7× to 12.5× at one support cell, p. 21), yet loses raw retrieval rankings every time to the noisy unshrunk average (Table E.1, p. 22). Inverting our explicit unclipped map on positive-weight coordinates without test labels restores the score within 10−310^{-3}, showing that lower raw retrieval reflects shrinkage rather than erased directions. Takeaway: improving accuracy depresses raw retrieval, which is why recent benchmarks rescale predictions before scoring.
  2. Check 1, constant-prediction floor (Section 4): Before crediting any model, feed an identical dummy prediction for every candidate. Under the official scorer, any rule that reads nothing specific to the evaluated perturbation and breaks ties independently of it must score exact chance, (N+1)/(2N)(N+1)/(2N). Yet matching each prediction to its target's true magnitude violates that condition and lifts a constant dummy input by up to +0.165 (to about 0.76 on RPE1, Table 1) purely from test-set norms. Takeaway: audit the scorer on a constant input first.
  3. Check 2, matched direction shuffling (Section 5): Hold each target's magnitude fixed and shuffle predicted directions across all 78 archived model-context pairs spanning 13 methods (scGPT, Geneformer, UCE, CPA, GEARS, scBERT, and six linear baselines), requiring no inverse or retraining. Under own-truth L1, 27 of 31 mean-beating pairs clear the global direction-null (25/31 under magnitude strata, Tables 2 and M.1), whereas scBERT's effectively constant archive outputs and GEARS on 384-target RPE1 split 2 beat the mean without clearing shuffled directions. Takeaway: beating a mean baseline is insufficient without direction shuffling.
  • ICLR contribution & where to view the restructured text: Rather than flagging two isolated models, these checks give ML benchmarkers a lightweight, model-agnostic audit for retrieval leaderboards wherever shared responses and shrinkage arise (recommending raw PDS alongside prediction error, top-effect recovery, and direction nulls). This restructured narrative is already live in the 70-page PDF uploaded on 15 November via the PDF icon beside the title (changes in blue): Sections 3–5 (pp. 3–8) present the core black-box audit and four-step procedure (p. 7), and Section 6 (pp. 8–9) condenses the supporting shrinkage study.

I thank the authors for their further comment, which is much clearer. As I indicated in my earlier response, the extent of the modifications and restructuring mentioned go beyond what a reviewer can reasonably assess over a rebuttal period, so I will not be modifying my recommendation. I would encourage the authors to resubmit a revised version to a future venue

Reviewer_v8BQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits where perturbation discrimination scores (PDS) get their perturbation-specific information. Giving every perturbation the same prediction yields exactly (N+1)/(2N) for any rule that reads nothing row-specific; rules that read the receiving perturbation’s own truth (the “fixed norm” of Liu et al., own-response weights, own top-k DE genes) move off the floor in either direction. Matched permutations (bundle vs direction-only) applied to 270 internal prediction sets and 78 published model–context pairs show that most published gains over the mean reflect real directional information, with GEARS on one RPE1 split and scBERT (effectively constant predictions) as exceptions. For empirical-Bayes shrinkage, inverting the known map restores the unshrunk ℓ2 score within 10⁻³ in all 90 settings, whereas scalar rescaling recovers between 95% and 1% of the gap depending on shared-response dominance.

Critical strengths

1- Proposition 2 plus Table 1 is a sharp, cheap diagnostic. Showing that the published fixed-norm implementation reads each perturbation’s true norm, and that this alone lifts an identity-free prediction by up to 0.165, is a concrete correction to a recommendation people are following. The scBERT result (constant predictions beat the mean under own-truth normalization) is a good illustration.

2- The paper is careful to a fault — pre-registered criteria, reported failures (ℓ1 optimum search, training-only map fit), tie-rule verification against the official scorer.

Critical weaknesses

1- Much of the length is spent on the authors’ own EB shrinkage estimator. The known-map inverse is, as they say, a positive control; it tells us nothing about any deployed model, and the ρ_shared ordering is five panels with an unstable middle. Sections 6–7 and roughly half the appendix could go without weakening the actual contribution.

2- The paper identifies the problem but doesn’t give a usable recommendation beyond “report matched nulls and declare inputs.” Matched-null ranges are randomization references, and the authors explicitly decline to define a null-subtracted metric. A reader finishing this paper knows what not to trust but not what score to report.

Questions for authors
  • Is there a single deployable correction you’d recommend for the Virtual Cell Challenge setting? Support-norm correction reads legitimate information but worsens rpe1 at every k.

  • For the 78 published pairs, how sensitive are the A/B/C classifications to the permutation seed? You note three cases within 10⁻⁴ of a boundary.

  • Why not report the identity-free control and matched null for the VCC leaderboard entries directly?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer v8BQ

For the evaluated VCC setting (cell-eval 0.7.2 L1), we recommend raw L1 PDS alongside response error, an effect-recovery endpoint and the matched direction-null. We have not found a uniformly preferable deployable normalization. Identity-free controls and matched nulls locate a score's information source for predictors aligned with the candidates. Known-map inversion is a supporting positive control.

1. Which correction to report. Support-norm matching needs support cells and worsens RPE1 at every tested support size (median L2 gap recovery -48.9% on eligible settings, Tables R.1, S.1). Training-fixed L1 lowers raw L1 on RPE1 by up to 0.0671 on common and 0.0822 on full libraries (Tables L.12-L.13). Predeclared training-only normalization or cosine can be sensitivity checks, and own-truth normalization a truth-conditioned diagnostic. Each needs declared inputs, an identity-free control and a matched direction-null. We define no null-subtracted metric. In the L2 RPE1 k=1 example, the oracle-norm identity-free lift of 0.1574 and oracle recovery of 0.1256 on real predictions concern different inputs (Table S.3), so one is not an identified additive share of the other. Table M.1's median null-share of -3% is descriptive.

On six common libraries reusing three screens (Appendix Z.2), raw-to-training-fixed and raw-to-own-truth L1 reverse 7/36 and 9/36 point orderings, with 0 and 1 having opposite-direction unadjusted 95% intervals conditional on fixed predictions and libraries. Across 78 pairs (Appendix Z.3), PDS gains and top-20 effect-recovery gains have correlations of 0.49 raw, 0.57 training-fixed and 0.55 own-truth. Their overlapping intervals resample pairs within contexts, condition on three screens and ignore related-method dependence. Within the same pairs, per-perturbation PDS scores versus top-20 overlap have median correlations of 0.59 training-fixed, 0.50 own-truth and 0.22 raw, without intervals. Both endpoints use the same measured truth. These associations do not select a biologically preferable rule.

2. Sensitivity to the permutation seed. After reproducing all 468 original rule-by-reference cells, we repeated the 78-pair family under 20 independent streams (Appendix Z.1), with 2,000 permutations per stream across three L1 rules and global and stratified direction-nulls, fixing predictions, libraries and the bootstrap mean-gain criterion.

Here 464/468 cells and all stratified own-truth classes are invariant, with GEARS RPE1 split 2 (384 targets) and both scBERT K562 cases inside their own-truth nulls throughout. Only four cells vary, the three 10⁻⁴ global boundary cases in Appendix M and one stratified endpoint case.

Pair, rule/nullOver 20 streams
Random-pert. linear, RPE1 split 1, own-truth/globalInside 18, above 2
GEARS, RPE1 split 2, raw/globalBelow 19, inside 1
scGPT, Adamson split 1, training-fixed/globalInside (at upper endpoint) 4, strictly above 16
RPE1-emb. linear, Adamson split 2, training-fixed/stratifiedInside 17 (16 at endpoint), above 3

Exact rank comparisons avoid counting floating-point ties as separation. These streams test Monte Carlo sensitivity, not biological replication.

3. A separate numerical correction. On Adamson split 2 (N=13), recomputing the 2,000 bootstrap draws on the exact PDS rank lattice moves the RPE1-embedding linear model's own-truth mean-gain lower bound from 8.54e-18 to zero, failing the strict-positive criterion. Auditing all 234 criteria found no other change.

Own-truth mean-beaters fall from 32 to 31, giving original-seed A/B/C counts of 4/27/47 globally and 6/25/47 under strata (27 and 25 above null), while raw and training-fixed counts stay 31 and 32. Table 2's B* entry becomes C. In Table M.1, own-truth inside shares become 13% and 19% and median null-share −3% (was −1%). Own-truth non-beaters above the global null become 8/47 (was 7/46), and mean-beaters entering the stratified range become 2 (was 3). This comes from exact rank arithmetic, not permutation seeds.

4. VCC entries and structure. Under row-independent official scoring, Proposition 2 fixes the identity-free floor at (N+1)/(2N). Applying matched nulls to leaderboard entries requires predictions aligned to the competition library, gene space and scorer version. That complete set was unavailable (Section 7), so we make no leaderboard-specific claim.

The revision puts the identity-free and matched-null analyses first, defines terms at first use and merges former Sections 6-7 into a shorter supporting Section 6, retaining the full results in the appendices. Known-map inversion does not correct black-box models. The five-panel L2 scalar ordering, unstable in its middle under resampling, does not identify a biologically correct ranking.

Thank you! this addresses what I asked. The 20-stream check settles the seed-sensitivity question, and I appreciate the self-reported lattice correction. I read your answer to the first question as confirming that no deployable normalization is uniformly better, which is a fair conclusion. The restructuring you describe is the right direction, but I don’t see a revised manuscript on the submission page, so I’m evaluating the paper as submitted and keeping my recommendation; my remaining concern is presentation rather than substance.

AuthorsOpenReview ↗

Pointer to Uploaded Revised Manuscript (15 Nov 2026) and Presentation Map

Thank you for the prompt follow-up and for confirming that the 20-stream seed check, the exact rank-lattice audit, and the normalization conclusion address your substantive questions.

A revised 70-page manuscript (with added and rewritten text marked in blue) was uploaded via Rebuttal Revision on 15 Nov 2026 prior to our initial response. Because OpenReview keeps the same top-level PDF URL (https://openreview.net/pdf?id=f49tlZwh7y) and the portal HTML abstract was left unchanged, browsers that opened the original 68-page submission can serve a cached copy unless hard-refreshed (Ctrl/Cmd+Shift+R) or accessed via the Revisions history link at the top of the page (https://openreview.net/revisions?id=f49tlZwh7y).

For convenience, here is where the presentation changes you raised appear in the uploaded 70-page blue-marked PDF:

  • Front-loaded core diagnostic and plain-language definitions (Sections 1 and 3–5, pp. 1–8): Sections 3–5 place the black-box scoring-input controls ahead of the empirical-Bayes case study, remove gauge-group terminology from the main text, and define tie-optimistic scoring, global versus per-prediction scaling, bundle nulls, and direction nulls at first use (Sections 3–5, pp. 4–6).
  • Actionable reporting checklist (Section 5, pp. 7–8): "Using the controls" (p. 7) states a concrete four-step reporting workflow for black-box evaluations (declaring rule inputs, running the identity-free floor, running matched permutations, and reporting raw ℓ1\ell_1 PDS alongside response error and an effect-recovery endpoint), followed by "Scoring sensitivity and effect recovery" (pp. 7–8).
  • Merged and shortened empirical-Bayes section (Section 6, pp. 8–9): Former Sections 6 and 7 are merged into a single compact supporting section ("Case study: recovering a known shrinkage map"), explicitly framed in its opening sentence as a positive control on our own estimator rather than a black-box correction, with detailed derivations and five-panel ρshared\rho_{\mathrm{shared}} stability analyses retained in the appendices.
  • Lattice update and new analyses (Table 2, p. 7; Appendix Z, pp. 69–70): The PDF abstract and Table 2 reflect the exact-lattice counts (31 and 27), and Appendix Z reports the 20-stream permutation check (Z.1), common-library rankings (Z.2), and top-20 effect recovery (Z.3).

We hope these pointers make it straightforward to inspect the restructured manuscript, and we welcome any further presentation suggestions during the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief Closing Summary Before Discussion Ends

Thank you to all three reviewers and the Area Chair for the constructive discussion. In our follow-up responses and today's updated PDF (changes highlighted in blue in Sections 1, 6, and 8), we clarified how our matched permutations separate magnitude alignment from directional discrimination across 78 published model-context pairs, why gene-wise empirical-Bayes shrinkage rotates predictions in a way a single scalar cannot undo (providing a known-map positive control for metric attribution), and how our black-box controls require no map inversion. As the discussion period wraps up, please let us know if any remaining question can be clarified.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.