acceptodds
Under review as a conference paper at ICLR 2027

Do System Gains Reflect Useful Predictive Variation?

Abstract

Scientific prediction systems can reduce error by correcting an overall offset, removing harmful baseline variation, or predicting useful differences between samples. How much of an observed gain comes from predicting sample differences? We compare fixed drug-response and histology predictions with controls that remove sample variation while retaining the reference adjustment. In four of 28 drug comparisons, the system improves on the adjusted baseline even though its added predictive variation increases error; removing the old variation accounts for the gain. In histology, source-fitted amplitude changes contribution without changing gene-wise correlation, and risk weighting changes the judgment of an identical output. Downstream Hallmark scoring introduces a further distinction. Across five models, 92 patients, and 50 gene sets, moving removal from expression to final scores reduces the number of positive simultaneous intervals from 39 to 14 out of 250 comparisons. The gap arises because the two operations produce constant controls with different means, changing the loss against which the same prediction is judged. These findings separate the value of a complete system improvement from the contribution of the sample differences it predicts.

Primary area:
datasets and benchmarks
Keywords:
model evaluation, predictive variation, scientific machine learning, calibration, downstream evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CFj7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks how much of a model’s gain over a baseline comes from predicting real differences between samples. The alternatives are fixing an overall offset or discarding a poor baseline’s variation. The authors compare each fixed prediction F with a control C that removes sample-to-sample variation but keeps the reference adjustment, and define the contribution P = L(C) − L(F) (Eq. 1). A telescoping identity splits the total gain into reference adjustment R, removal of old baseline variation H, and added model variation P (Eq. 2, 4). P is further split into alignment minus amplitude cost (Eq. 6).

Critical strengths
  1. The analysis seems careful, verifiable, and well written.
  2. Section 5 provides good guidance for anyone computing gene-set scores from predicted expression. To measure the value of patient-level differences, the variation should be removed after scoring.
Critical weaknesses

1 The paper’s core tools are established ones, and the paper doesn’t position itself against them. 1.1 P (Eq. 1) is the numerator of a skill score against a constant reference forecast, a standard construction in forecast verification (Murphy, 1988). Any novelty lies in the choice of reference, not the measure itself, and the paper does not compare its reference against standard alternatives.

1.2 The alignment–amplitude split (Eq. 6, App. C.2) matches the correlation and amplitude terms of Murphy’s (1988) MSE skill decomposition; the bias term drops out because F and C share the same mean. The finding that amplitude changes the error but not the PCC (Sec. 4.2, Fig. 2) follows directly. Sec. 6 calls Murphy “complementary,” which understates the overlap.

1.3 The reference adjustment and fitted amplitude (Eq. 3, App. A.1, A.4) amount to intercept and slope recalibration, related to Mincer–Zarnowitz regression in forecasting and to calibration-in-the-large and calibration slope in clinical prediction (e.g., Van Calster et al., 2019).

1.4 Score decompositions into miscalibration, discrimination and uncertainty (Dimitriadis et al., 2021; Gneiting & Resin, 2023; Arnold et al., 2024) already quantify the useful predictive variation that remains after recalibration, which is the paper’s central question. These are not cited.

1.5 The headline empirical messages echo prior benchmarks. DrEval (Bernett et al., 2026) showed that drug-response models barely beat a naive model of mean drug and cell-line effects, which is essentially the baseline b used here (Sec. 3.1). Systema (Viñas Torné et al., 2026) made a similar point for perturbation prediction. The H vs. P split adds detail but not a new conclusion.

NOTE: These references were found via LLM assisted tools. I do stand by my critique of lack of novelty

Questions for authors

P (Eq. 1) is the numerator of a skill score against a constant reference forecast (Murphy, 1988). Its novelty therefore rests on the choice of reference C. Could you clarify: (a) why the reference-adjusted constant (Eq. 3, App. A.1) is the right reference, rather than the outcome mean or a recalibrated prediction; (b) whether any conclusions in Table 1, Table 3 or Table 6 change if C is replaced by the evaluation outcome mean, as in R² or Nash–Sutcliffe efficiency; and (c) what P, H and Δ reveal that the discrimination term of Gneiting & Resin (2023) does not?

8 comments

Thank you. We revised the positioning and added the requested reference comparisons.

We provide a revision summary in the top-level comment and a marked-up manuscript in the supplementary ZIP (manuscript_highlight.pdf).

1.1–1.3. Established tools.

P=L(C)−L(F)P=L(C)-L(F) is a skill-score numerator. Equation (6) uses Murphy's correlation/scale decomposition; shared mean bias cancels. Reference adjustment and amplitude fitting are intercept- and slope-type recalibration. Sections 4.1 and 6 state these correspondences, including Mincer–Zarnowitz regression and clinical calibration. Our histology results quantify transferred loss consequences of source-fitted amplitudes and risk weighting; correlation's invariance to positive scaling is established.

(a) Why this constant?

For each output, C0=F‾qC_0=\overline F_q is the unique constant preserving the prediction mean under normalized evaluation weights q (Section 2.2; Table 1). In the drug construction, F‾q=b‾q+δ\overline F_q=\overline b_q+\delta, retaining the reference shift. It asks whether retaining sample variation lowers loss at the same mean, and requires the unlabeled evaluation batch (Section 7). The outcome mean CY=Y‾qC_Y=\overline Y_q measures skill against the best evaluation-sample constant under squared loss; recalibration evaluates gains from altering the forecast. Tables 24–26 compare constants with complete predictions fixed.

1.4 and (c). Relation to discrimination.

We added Dimitriadis et al. (2021), Gneiting & Resin (2023), and Arnold et al. (2024). With identical loss and weights, Eq. (7) gives

PC=DSC−MCB+L(C)−UNC.P_C=\mathrm{DSC}-\mathrm{MCB}+L(C)-\mathrm{UNC}.

Under squared loss, PC0=DSC−MCBcP_{C_0}=\mathrm{DSC}-\mathrm{MCB}_c, where MCBc\mathrm{MCB}_c is miscalibration after mean-bias correction (Appendix J.2). COAD/DeepPT has DSC=7.671 pp but P=-5.744 pp under gene-macro risk. The 45 histology diagnostics (Tables 29–31) use evaluation labels for within-sample isotonic fitting. H also needs baseline B and control C: H=MCBB−DSCB+UNC−L(C)H=\mathrm{MCB}_B-\mathrm{DSC}_B+\mathrm{UNC}-L(C). For downstream constants a=g(F‾)a=g(\overline F) and b=g(F)‾b=\overline{g(F)}, Δ=(a−m)2−(b−m)2\Delta=(a-m)^2-(b-m)^2, with observed score mean m. DSC of F alone specifies neither comparison. Full existing decompositions recover these quantities given the same inputs.

(b) Outcome-mean sensitivity: original Tables 1, 3 and 6.

Yes, conclusions change. These original tables are now Tables 2, 3 and 7; Tables 24–26 give the added comparisons.

  • Drug response: under the adaptation protocol in Section 3.1, positive/adverse/unresolved P intervals across 28 conditions are 8/8/12 for C0C_0, 21/4/3 for the unadjusted constant, and 0/20/8 for CYC_Y. All four original positive-J/adverse-P cases remain adverse against CYC_Y but become positive against the unadjusted constant, which also credits mean adjustment. J and S point values and paired draws remain unchanged. Each reference uses its declared simultaneous family.
  • Histology: the identical COAD/DeepPT output gives gene-macro/native-pooled/standardized P=-5.744/1.896/-0.699 pp against C0C_0. Against CYC_Y, P=-16.659/-13.145/-15.846 pp: all three intervals are adverse, replacing the original adverse/positive/unresolved classifications. Against the unadjusted constant all three are positive. Table 24 uses one nine-coordinate family.
  • Downstream: Table 26 retains all 25 original COAD cases; against the observed score mean, 18 are adverse and seven unresolved. The 750-coordinate reference family and original 765-coordinate removal-location family contain different, non-nested contrasts. Replacing both controls by the same outcome mean makes their gap zero by construction and answers the outcome-mean skill question. Equation (8) compares the two removal locations.

1.5. DrEval, Systema and the empirical contribution.

Section 6 describes DrEval's normalized R2R^2 and correlations after subtracting its mean-effects predictor b from predictions and outcomes. For ordinary weighted R2R^2 on identical support with positive residual variance, let L(f)=∑iqi(Yi−fi)2L(f)=\sum_iq_i(Y_i-f_i)^2. Then

Rq2(Y−b,F−b)=1−L(F)L(b+Y−b‾q).R_q^2(Y-b,F-b)=1-\frac{L(F)}{L(b+\overline{Y-b}_q)}.

This reference retains b's variation; C0C_0 preserves F's mean and removes its variation (derivation in Appendix J.2). Under our declared adaptation, 21 positive-J conditions comprise eight positive, nine unresolved and four adverse P intervals (Table 27). Joint reporting separates evidence for replacing the baseline from evidence for retaining the adapted sample variation. Section 6 also credits Systema's analysis of systematic variation. With complete downstream scores fixed, added BRCA/GBM analyses show 43/38 positive-to-unresolved changes across removal locations, with 41/38 positive paired control-loss intervals (Table 4). These results quantify the consequences of the output comparisons and motivate Section 7's reporting guidance.

Reviewer_CFj7OpenReview ↗

Response to Rebuttal

Dear Authors,

Apologies for the elongated suspense. I have read your rebuttal carefully and used LLM to help structure this reply, to check the validity of my own arguments and educate myself further so i can make an informed decision. The assessment is my own. I am using the following criteria to judge your rebuttal.

Does it strengthen the paper’s key claims? Does it respond to my review? Does it contradict earlier arguments or evidence?

For 1: Partly. The BRCA/GBM extension and source-resampling analyses strengthen the empirical side. However, the new reference comparisons (Tables 24–26) show that the headline findings depend on choosing C₀. Under the outcome mean, no drug condition has a positive P, and the COAD/DeepPT verdicts become uniformly adverse. Against the unadjusted constant, the four highlighted drug cases and all three histology risks turn positive. The findings are therefore properties of the chosen control rather than of the predictions themselves.

For 2: Yes, thoroughly. You now state that P is a skill-score numerator, that Eq. 6 is Murphy’s decomposition, that the adjustments are intercept/slope recalibration, and that P_C₀ = DSC − MCB_c. I appreciate the candour, but this confirms rather than resolves my novelty concern, since existing decompositions recover your contrasts given the same controls. The DrEval distinction is useful.

For 3: No direct contradictions, but two tensions remain. Section 6 still calls Murphy “complementary,” although Eq. 6 is his decomposition. The abstract also presents risk weighting as changing the verdict on an identical output in general, while the new tables show this holds only under C₀.

Overall: The paper would be stronger if reframed around reference dependence as its main finding, with C₀ motivated as one deliberate choice. As it stands, I see it as an incremental empirical application of established tools, and I maintain my score of 2.

Dear Reviewer CFj7,

Thank you for following up and for suggesting the comparison with the outcome mean. We agree that established decompositions recover the loss contrasts once the controls are specified. Our empirical contribution is to establish where these distinctions materially change the conclusions supported by the evaluated scientific prediction systems. The established identities explain how such differences can arise; the experiments determine their direction, magnitude, and consequences for claims about useful sample-specific prediction.

To clarify the version history, the wording “complementary tools” appears in §6 of the original submission (p. 8). Your first review prompted us to revise that passage. In the manuscript accompanying our 14 November rebuttal, §4.1 (p. 5) identifies the two terms of Eq. (6) as the correlation and scale components of Murphy’s MSE decomposition, and §6 (p. 9) states that our alignment–amplitude split “uses the same geometry.”

We agree with your reading of Table 24. The risk-weighting result in the abstract refers to the mean-matched comparison for COAD/DeepPT. With the predictions fixed, the contribution intervals are negative, positive, and unresolved under gene-macro, native-pooled, and standardized risk, respectively, against C0C_0. All three are negative against the outcome mean you suggested and positive against the unadjusted constant. These results delimit the claim: the effect of risk weighting is reference-specific, and the different references assess different aspects of the same output. The sentence immediately following this result in the revised abstract introduces the comparison across references.

In the drug study, we chose C0C_0 because it is the unique constant that preserves each drug’s weighted prediction mean (§2.2). It therefore addresses the specific question of whether retaining the predicted cell-line differences reduces loss while keeping that mean fixed. Of the 21 conditions in which the adapted system improved on the adjusted baseline, eight favored retaining those differences, four favored using the per-drug means, and nine were unresolved, according to the simultaneous intervals (Table 27). On GDSC1, for example, the adapted SNN lowered normalized loss by 4.25 percentage points relative to the adjusted baseline. Replacing its cell-line predictions with their per-drug means lowered loss by a further 1.49 points (Fig. 1c).

This result changes the attribution of the observed improvement: the system benefits from removing harmful baseline variation even though its newly added cell-line variation increases error. A positive replacement gain consequently supports a different conclusion from a positive contribution of predicted sample differences. Among the three references, the four cases you discuss become positive only against the unadjusted constant, which also credits the mean adjustment. The four SRMF cases remain negative under all three references (Table 25).

Your first review also highlighted the practical importance of where variation is removed when evaluating patient differences in gene-set scores. The two added cohorts quantify the consequence of that choice. With the predicted Hallmark scores and their final MSE held fixed, moving removal from gene expression to the final scores changed 43 model–gene-set comparisons in BRCA and 38 in GBM from positive to unresolved. Positive paired control-loss differences support 41 of those 43 changes in BRCA and all 38 in GBM (Table 4). Thus, the results establish that control placement can materially change the evidence supporting the benefit of predicted patient-to-patient score variation. Preserving the predicted score mean isolates that contribution from the additional effect of moving the constant reference. “Unresolved” here indicates insufficient evidence to determine the contribution’s direction, rather than evidence of ineffectiveness.

These findings provide concrete limits on what observed performance improvements establish about sample-specific prediction. They distinguish evidence for replacing a baseline from evidence for retaining the replacement’s predicted sample differences, and quantify when downstream control placement changes the latter judgment. We would be grateful if you would reconsider the empirical contribution and your overall assessment in light of these consequences for interpreting scientific prediction gains.

Reviewer_iLEdOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper explores how much of the system gain in predictive systems is attributable to predicted sample-to-sample variations. The authors study this problem by removing baseline variation, changing the amplitude of the prediction, and changing the location of variation removal for downstream scoring. They conduct empirical analyses across three use cases (drug response estimation, gene expression prediction from histology, and precipitation forecasting) to answer this question. Through their analyses, they demonstrate that: (i) the system gain can be driven by removing harmful variations in the baseline, even when the newly added model variation increases error; (ii) changing the amplitude of the prediction can change its contribution to the system gain while preserving correlation; and (iii) the estimated contribution of downstream predictions can change depending on where the variations are removed.

Critical strengths

The paper systematically explores an important question regarding the source of system gain by conducting empirical analyses in sensitive use cases where such perspectives matter greatly.

The comparisons are sound, with key experimental conditions fixed, which makes the results easier for the audience to interpret.

The key findings listed in the summary above are interesting and could potentially change how we design and interpret benchmarking experiments in these domains.

The analysis through multiple lenses (baseline variation removal, amplitude changes, and variation removal location) adds depth to the analyses and strengthens the results.

Critical weaknesses

Despite the multiple strengths, I am leaning towards weak accept because I would like to see the points below addressed.

The broader implication of the results for these domains should be elaborated upon. What the findings could mean for future experimental design and benchmarking? This information is currently diluted in the introduction and the discussion. A concrete guidance would be beneficial for the readers from the respective domains.

Add a limitations section highlighting the following points: (i) robustness to retraining and refitting amplitude across different seeds needs to be discussed, (ii) downstream validation is restricted to singscore on COAD; broader applicability tests can be future work.

Questions for authors

Define b‾\overline{b} and p‾\overline{p} in equation 3.

It would be nice to mention how original baseline bib_i is calculated.

Section 4.1 Define Y and w in the main text.

Section 5.3: Define the terms in the equations here more intuitively for easy interpretation. Defer the detailed explanations to the appendix.

3 comments

Thank you for the constructive suggestions. We have addressed the practical implications, expanded the downstream evidence, and clarified the uncertainty and notation.

We provide a revision summary in the top-level comment and a marked-up manuscript in the supplementary ZIP (manuscript_highlight.pdf).

1. Guidance for experimental design and benchmarking.

Section 7 now gives task-specific recommendations. Drug benchmarks should state the baseline construction and reference adjustment and report replacement gain J jointly with variation contribution P. For expression prediction, report the risk weights, normalization and fitted amplitude alongside correlation. For patient-level gene-set evaluation, compute complete scores first, then remove patient variation while preserving the predicted score mean; report the paired difference from expression-level removal when that intervention is relevant. Proposed recalibration should be fitted on source or held-out calibration data. These choices connect the reported gain to the prediction property being claimed.

2. Amplitude refitting and retraining uncertainty.

We added 1,000 source-patient resamples per cohort, refitting the original-scale and standardized amplitudes while holding source out-of-fold predictions, preprocessing estimates and target data fixed. Of 90 fitted-output/risk conditions, 82 retain their contribution point sign in every resample; the remaining eight have 88.3–99.7% sign stability (Section 4.2; Appendix J.4, Tables 33–38).

This analysis measures source-sample sensitivity for these two amplitude fits. Section 7 states that stability of the full training procedure requires independent multi-seed retraining and re-estimation of preprocessing. The centered risk-matched fit and its paired improvement are outside this resampling analysis, as stated next to the result.

3. Downstream coverage.

The same singscore/Hallmark comparison now covers BRCA, COAD and GBM. The number of positive contribution intervals before versus after moving removal to the score level is 39 to 14 in COAD, 90 to 47 in BRCA, and 165 to 127 in GBM. Among the 25, 43 and 38 positive-to-unresolved changes, respectively, 24, 41 and 38 have positive paired control-loss intervals (Table 4; Appendix J.3). Each cohort has its own 765-coordinate simultaneous family. Section 7 retains the limitation to singscore and these Hallmark sets; testing other scoring methods is future work.

4. Definitions around original Eq. (3), now Eq. (4), and the original baseline.

Section 3.1 now defines all quantities together: bib_i is the released baseline prediction, pip_i the original model output, δ\delta the reference shift, and α\alpha the amplitude fitted on source out-of-fold predictions. The means b‾\overline b and p‾\overline p use the drug's common evaluation-cell support and are recomputed with each draw's weights. The DrEval naive mean-effects baseline is bcd=μ+ηc+γdb_{cd}=\mu+\eta_c+\gamma_d, with training grand mean μ\mu, cell effect ηc=y‾c−μ\eta_c=\overline y_c-\mu, and drug effect γd=y‾d−μ\gamma_d=\overline y_d-\mu; unavailable effects are zero. Our comparisons use the released baseline arrays.

5. Y and w in Section 4.1.

Y denotes observed responses. The nonnegative weights w include sample averaging, output aggregation and the declared risk normalization. Both losses use these same weights in Eq. (6).

6. Intuition for Section 5.3.

The text now defines a as the score of mean predicted expression, b as the mean predicted score, and m as the mean observed score; d=a-b is the control's movement and e=b-m its original mean error. Equation (10) compares the controls' squared distances from m. The text explains how this movement can increase or decrease control loss. Derivations remain in Appendix C.3, with the case-level decomposition in Appendix E.1.

Thank you. I am satisfied with the responses. I appreciate the extra effort put into addressing my concern about the limitations. The authors went beyond simply acknowledging these limitations by providing additional empirical results for downstream coverage and source-sample sensitivity. I will revise my score to 4. I am also happy to assist the AC with further evaluation of the paper during the next discussion phase.

Thank you for your thoughtful feedback and continued engagement. Your comments helped us strengthen the downstream evaluation and source-sample sensitivity analysis, and clarify the scope of our conclusions. We appreciate your willingness to participate in the upcoming discussion.

Reviewer_qRH2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the cases when a model reports improvement over a baseline. They attribute the improvement to the following categories: fixing an overall offset, fixing a baseline’s harmful sample-to-sample differences, or the case where the model is correctly distinguishing between samples.

The authors apply this framework to analyzing drug responses and histology. They demonstrate that in 4/28 cases a drug response model beats the standard baseline while being worse than a flat guess. The gain comes from removing the baseline’s variation. For histology, they find that predictions with identical correlations can raise or lower error depending on how spread out they are. The same predictions get positive, negative, or unclear verdicts depending on how the genes are weighted. For downstream hallmark scores, removing variation after as opposed to before scoring changes 25 of the 39 positive contributions to inconclusive. Regarding validation, the authors employ paired bootstrap with simultaneous intervals, held out cell deletions, and an analytic toy model for confirmation.

Critical strengths

The authors identified an important problem in the field; headline numbers tend to obfuscate the tradeoffs between different models on the distribution of data. The problem framing is consequential to biology, considering the transition from model nominations to wet lab validation.

Critical weaknesses

The contribution over existing tools is unclear. For example, the main “telescoping identity for any common scalar loss” by itself does not establish a novel evaluation method. The authors do not explain what additional conclusions their comparison provides in addition to existing tools, including the cited Murphy’s skill decomposition. The authors distinguish baseline-removal gains from added predictive variation and compare removal before and after downstream scoring. However, it is difficult to follow how these additions advance the methodology, and what the core novelty is.

Questions for authors
  1. Clarification: Which parts are established decomposition, which parts are new, and what consequential insight is given by the added design?
  2. Could you point to a concrete conclusion from your experiments that existing tools or baseline comparisons would not reveal, and explain which parts of your framework make that conclusion possible?
2 comments

Thank you for recognizing the biological importance of distinguishing system gains from evidence for sample-specific prediction. We have clarified the contribution and its relationship to established tools.

We provide a revision summary in the top-level comment and a marked-up manuscript in the supplementary ZIP (manuscript_highlight.pdf).

Q1. Which parts are established, and what does the comparison design add?

The loss difference P=L(C)-L(F) is a skill-score numerator, the telescoping identities follow by adding and subtracting losses, and the alignment–amplitude split uses Murphy's correlation/scale decomposition. Reference adjustment and amplitude fitting are intercept- and slope-type recalibration. Sections 4.1 and 6 now state these correspondences directly.

The contribution is the choice and joint evaluation of concrete output operations: replacing an existing baseline, retaining the model's sample variation at its current mean, and removing variation before versus after a downstream readout. Section 2.2 defines C0=F‾qC_0=\overline F_q, the unique sample-constant control preserving the prediction's weighted mean. Existing decompositions recover these contrasts once the same controls are supplied. The empirical contribution is to identify and quantify how these choices change the conclusions supported by the evaluated predictions.

Q2. What consequential conclusion does this reveal beyond a baseline comparison?

Reference sensitivity. The added analysis holds the complete predictions fixed under the adaptation protocol in Section 3.1. All four original positive-J/adverse-P cases have positive P intervals against the unadjusted constant, but retain adverse intervals against the outcome mean (Tables 24–25). The contrast against the unadjusted constant combines mean adjustment with sample variation; PC0P_{C_0} preserves the prediction mean and isolates the loss contribution of the retained variation. Thus the reference comparison determines which interpretation of a gain the evaluated outputs support. Each reference uses its declared simultaneous interval family.

Downstream scope. The added BRCA and GBM analyses find 43 and 38 positive-to-unresolved changes when removal moves from expression to final scores. The paired control-loss intervals are positive in 41 and 38 of these cases, respectively (Table 4). These extend the original COAD finding of 25 changes, with 24 positive paired differences. Each cohort uses its own simultaneous family. The complete scores and their errors stay fixed: the paired tests identify a difference between the two controls, and the score-level control preserves the complete predictions' mean score. Equation (10) relates the gap to the controls' distances from the observed score mean.

Consequence for evaluation. The original GDSC1/SNN example illustrates the distinction under the same adaptation protocol: removing baseline variation lowers normalized loss by H=5.738 pp, while adding model variation increases it by 1.492 pp, leaving positive replacement gain J=4.246 pp. The 95% simultaneous intervals are [0.37, 8.12] pp for J and [-2.02, -0.97] pp for P (Table 13). The output improves on the adjusted baseline, while a constant with the same prediction mean has lower loss still.

Jointly reporting J and P therefore separates evidence for replacing a baseline from evidence for retaining the evaluated sample differences. For patient-level gene-set variation, remove variation after scoring while preserving the predicted score mean; report the expression-level intervention separately when relevant, together with the paired control gap. Section 7 connects these reporting choices to the scientific claim. The new reference and cohort analyses quantify the sensitivity and scope of these conclusions.

Dear Reviewer qRH2,

Thank you again for your thoughtful review and for raising important questions about the paper's contribution.

As the discussion period draws to a close, we wanted to kindly follow up on our rebuttal submitted on 14 November. We would greatly appreciate any further thoughts you may have, particularly whether our response has addressed your main concerns or whether any questions remain.

We understand that you may have a busy schedule, and we sincerely appreciate the time and effort you have devoted to reviewing our work. Your feedback is valuable to us, and we would be grateful if you could take our response into account when finalizing your assessment.

Thank you again for your time and consideration.

Comments on the paper

1 comment

The table below summarizes the revisions and their locations.

Section and table numbers below refer to the revised manuscript. The scientific question and fixed-prediction comparisons are retained from the original submission.

Reviewer concernRevisionLocation
qRH2 Q1; CFj7 1.1–1.4: relationship to established toolsState the skill-score, Murphy and intercept/slope-calibration correspondences; connect P to the MCB/DSC/UNC decomposition, including conditional miscalibration under mean matching.Sections 4.1–4.2 and 6; Eq. 7; Appendix J.2
CFj7 (a): justification of the constantDefine the unique constant preserving the prediction's weighted mean and distinguish the questions answered by alternative constants and recalibration.Section 2.2; Table 1
CFj7 (b): sensitivity of original Tables 1, 3 and 6Add fixed-output reference comparisons, retaining all 28 drug conditions, the three-risk COAD/DeepPT example, and all 25 original downstream cases.Table 24 summary; Tables 25–26 complete drug/original-case results; Appendix J.1
CFj7 (c): relationship to discriminationAdd 45 histology isotonic diagnostics; express P, baseline-removal H and the downstream control gap using classical decomposition terms or control losses.Eq. 7; Appendix J.2; Tables 29–31
qRH2 Q2; CFj7 1.5: consequential findings and comparison with DrEvalExplain DrEval's residualized evaluation and its implicit reference; distinguish it from mean-preserving variation removal. Report reference sensitivity, three-cohort paired comparisons and the full J/P classification, identifying the decisions each comparison supports.Sections 2.1, 3.2, 5–7; Appendix J.2; Fig. 1; Tables 4, 24–27
iLEd: downstream scopeAdd BRCA and GBM analyses under the same singscore/Hallmark protocol, with cohort-specific simultaneous intervals and paired control-loss differences; summarize all three cohorts in the main text.Section 5.2; Table 4; Appendix J.3; Table 32
iLEd: amplitude refitting and retrainingAdd source-patient resampling for original-scale and standardized amplitudes. State the conditional scope and the remaining need for independent retraining.Section 4.2; Appendix J.4; Tables 33–38; Section 7
iLEd: practical implications and clarityAdd domain-specific reporting guidance; define the drug baseline and all output-construction quantities, Y and w, and the downstream control/mean terms in the main text.Sections 3.1, 4.1, 5.3 and 7

The supplementary ZIP contains an English marked-up manuscript at manuscript_highlight.pdf, highlighting changes relative to the original submission, and a copy of this table at revision_guide/REVISION_SUMMARY.md. Individual responses follow in the corresponding review threads.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.