- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks how much of a model’s gain over a baseline comes from predicting real differences between samples. The alternatives are fixing an overall offset or discarding a poor baseline’s variation. The authors compare each fixed prediction F with a control C that removes sample-to-sample variation but keeps the reference adjustment, and define the contribution P = L(C) − L(F) (Eq. 1). A telescoping identity splits the total gain into reference adjustment R, removal of old baseline variation H, and added model variation P (Eq. 2, 4). P is further split into alignment minus amplitude cost (Eq. 6).
Critical strengths
- The analysis seems careful, verifiable, and well written.
- Section 5 provides good guidance for anyone computing gene-set scores from predicted expression. To measure the value of patient-level differences, the variation should be removed after scoring.
Critical weaknesses
1 The paper’s core tools are established ones, and the paper doesn’t position itself against them. 1.1 P (Eq. 1) is the numerator of a skill score against a constant reference forecast, a standard construction in forecast verification (Murphy, 1988). Any novelty lies in the choice of reference, not the measure itself, and the paper does not compare its reference against standard alternatives.
1.2 The alignment–amplitude split (Eq. 6, App. C.2) matches the correlation and amplitude terms of Murphy’s (1988) MSE skill decomposition; the bias term drops out because F and C share the same mean. The finding that amplitude changes the error but not the PCC (Sec. 4.2, Fig. 2) follows directly. Sec. 6 calls Murphy “complementary,” which understates the overlap.
1.3 The reference adjustment and fitted amplitude (Eq. 3, App. A.1, A.4) amount to intercept and slope recalibration, related to Mincer–Zarnowitz regression in forecasting and to calibration-in-the-large and calibration slope in clinical prediction (e.g., Van Calster et al., 2019).
1.4 Score decompositions into miscalibration, discrimination and uncertainty (Dimitriadis et al., 2021; Gneiting & Resin, 2023; Arnold et al., 2024) already quantify the useful predictive variation that remains after recalibration, which is the paper’s central question. These are not cited.
1.5 The headline empirical messages echo prior benchmarks. DrEval (Bernett et al., 2026) showed that drug-response models barely beat a naive model of mean drug and cell-line effects, which is essentially the baseline b used here (Sec. 3.1). Systema (Viñas Torné et al., 2026) made a similar point for perturbation prediction. The H vs. P split adds detail but not a new conclusion.
NOTE: These references were found via LLM assisted tools. I do stand by my critique of lack of novelty
Questions for authors
P (Eq. 1) is the numerator of a skill score against a constant reference forecast (Murphy, 1988). Its novelty therefore rests on the choice of reference C. Could you clarify: (a) why the reference-adjusted constant (Eq. 3, App. A.1) is the right reference, rather than the outcome mean or a recalibrated prediction; (b) whether any conclusions in Table 1, Table 3 or Table 6 change if C is replaced by the evaluation outcome mean, as in R² or Nash–Sutcliffe efficiency; and (c) what P, H and Δ reveal that the discrimination term of Gneiting & Resin (2023) does not?
8 comments
Response to Rebuttal
Dear Authors,
Apologies for the elongated suspense. I have read your rebuttal carefully and used LLM to help structure this reply, to check the validity of my own arguments and educate myself further so i can make an informed decision. The assessment is my own. I am using the following criteria to judge your rebuttal.
Does it strengthen the paper’s key claims? Does it respond to my review? Does it contradict earlier arguments or evidence?
For 1: Partly. The BRCA/GBM extension and source-resampling analyses strengthen the empirical side. However, the new reference comparisons (Tables 24–26) show that the headline findings depend on choosing C₀. Under the outcome mean, no drug condition has a positive P, and the COAD/DeepPT verdicts become uniformly adverse. Against the unadjusted constant, the four highlighted drug cases and all three histology risks turn positive. The findings are therefore properties of the chosen control rather than of the predictions themselves.
For 2: Yes, thoroughly. You now state that P is a skill-score numerator, that Eq. 6 is Murphy’s decomposition, that the adjustments are intercept/slope recalibration, and that P_C₀ = DSC − MCB_c. I appreciate the candour, but this confirms rather than resolves my novelty concern, since existing decompositions recover your contrasts given the same controls. The DrEval distinction is useful.
For 3: No direct contradictions, but two tensions remain. Section 6 still calls Murphy “complementary,” although Eq. 6 is his decomposition. The abstract also presents risk weighting as changing the verdict on an identical output in general, while the new tables show this holds only under C₀.
Overall: The paper would be stronger if reframed around reference dependence as its main finding, with C₀ motivated as one deliberate choice. As it stands, I see it as an incremental empirical application of established tools, and I maintain my score of 2.
Dear Reviewer CFj7,
Thank you for following up and for suggesting the comparison with the outcome mean. We agree that established decompositions recover the loss contrasts once the controls are specified. Our empirical contribution is to establish where these distinctions materially change the conclusions supported by the evaluated scientific prediction systems. The established identities explain how such differences can arise; the experiments determine their direction, magnitude, and consequences for claims about useful sample-specific prediction.
To clarify the version history, the wording “complementary tools” appears in §6 of the original submission (p. 8). Your first review prompted us to revise that passage. In the manuscript accompanying our 14 November rebuttal, §4.1 (p. 5) identifies the two terms of Eq. (6) as the correlation and scale components of Murphy’s MSE decomposition, and §6 (p. 9) states that our alignment–amplitude split “uses the same geometry.”
We agree with your reading of Table 24. The risk-weighting result in the abstract refers to the mean-matched comparison for COAD/DeepPT. With the predictions fixed, the contribution intervals are negative, positive, and unresolved under gene-macro, native-pooled, and standardized risk, respectively, against . All three are negative against the outcome mean you suggested and positive against the unadjusted constant. These results delimit the claim: the effect of risk weighting is reference-specific, and the different references assess different aspects of the same output. The sentence immediately following this result in the revised abstract introduces the comparison across references.
In the drug study, we chose because it is the unique constant that preserves each drug’s weighted prediction mean (§2.2). It therefore addresses the specific question of whether retaining the predicted cell-line differences reduces loss while keeping that mean fixed. Of the 21 conditions in which the adapted system improved on the adjusted baseline, eight favored retaining those differences, four favored using the per-drug means, and nine were unresolved, according to the simultaneous intervals (Table 27). On GDSC1, for example, the adapted SNN lowered normalized loss by 4.25 percentage points relative to the adjusted baseline. Replacing its cell-line predictions with their per-drug means lowered loss by a further 1.49 points (Fig. 1c).
This result changes the attribution of the observed improvement: the system benefits from removing harmful baseline variation even though its newly added cell-line variation increases error. A positive replacement gain consequently supports a different conclusion from a positive contribution of predicted sample differences. Among the three references, the four cases you discuss become positive only against the unadjusted constant, which also credits the mean adjustment. The four SRMF cases remain negative under all three references (Table 25).
Your first review also highlighted the practical importance of where variation is removed when evaluating patient differences in gene-set scores. The two added cohorts quantify the consequence of that choice. With the predicted Hallmark scores and their final MSE held fixed, moving removal from gene expression to the final scores changed 43 model–gene-set comparisons in BRCA and 38 in GBM from positive to unresolved. Positive paired control-loss differences support 41 of those 43 changes in BRCA and all 38 in GBM (Table 4). Thus, the results establish that control placement can materially change the evidence supporting the benefit of predicted patient-to-patient score variation. Preserving the predicted score mean isolates that contribution from the additional effect of moving the constant reference. “Unresolved” here indicates insufficient evidence to determine the contribution’s direction, rather than evidence of ineffectiveness.
These findings provide concrete limits on what observed performance improvements establish about sample-specific prediction. They distinguish evidence for replacing a baseline from evidence for retaining the replacement’s predicted sample differences, and quantify when downstream control placement changes the latter judgment. We would be grateful if you would reconsider the empirical contribution and your overall assessment in light of these consequences for interpreting scientific prediction gains.
Thank you. We revised the positioning and added the requested reference comparisons.
We provide a revision summary in the top-level comment and a marked-up manuscript in the supplementary ZIP (
manuscript_highlight.pdf).1.1–1.3. Established tools.
P=L(C)−L(F) is a skill-score numerator. Equation (6) uses Murphy's correlation/scale decomposition; shared mean bias cancels. Reference adjustment and amplitude fitting are intercept- and slope-type recalibration. Sections 4.1 and 6 state these correspondences, including Mincer–Zarnowitz regression and clinical calibration. Our histology results quantify transferred loss consequences of source-fitted amplitudes and risk weighting; correlation's invariance to positive scaling is established.
(a) Why this constant?
For each output, C0=Fq is the unique constant preserving the prediction mean under normalized evaluation weights q (Section 2.2; Table 1). In the drug construction, Fq=bq+δ, retaining the reference shift. It asks whether retaining sample variation lowers loss at the same mean, and requires the unlabeled evaluation batch (Section 7). The outcome mean CY=Yq measures skill against the best evaluation-sample constant under squared loss; recalibration evaluates gains from altering the forecast. Tables 24–26 compare constants with complete predictions fixed.
1.4 and (c). Relation to discrimination.
We added Dimitriadis et al. (2021), Gneiting & Resin (2023), and Arnold et al. (2024). With identical loss and weights, Eq. (7) gives
PC=DSC−MCB+L(C)−UNC.
Under squared loss, PC0=DSC−MCBc, where MCBc is miscalibration after mean-bias correction (Appendix J.2). COAD/DeepPT has DSC=7.671 pp but P=-5.744 pp under gene-macro risk. The 45 histology diagnostics (Tables 29–31) use evaluation labels for within-sample isotonic fitting. H also needs baseline B and control C: H=MCBB−DSCB+UNC−L(C). For downstream constants a=g(F) and b=g(F), Δ=(a−m)2−(b−m)2, with observed score mean m. DSC of F alone specifies neither comparison. Full existing decompositions recover these quantities given the same inputs.
(b) Outcome-mean sensitivity: original Tables 1, 3 and 6.
Yes, conclusions change. These original tables are now Tables 2, 3 and 7; Tables 24–26 give the added comparisons.
1.5. DrEval, Systema and the empirical contribution.
Section 6 describes DrEval's normalized R2 and correlations after subtracting its mean-effects predictor b from predictions and outcomes. For ordinary weighted R2 on identical support with positive residual variance, let L(f)=∑iqi(Yi−fi)2. Then
Rq2(Y−b,F−b)=1−L(b+Y−bq)L(F).
This reference retains b's variation; C0 preserves F's mean and removes its variation (derivation in Appendix J.2). Under our declared adaptation, 21 positive-J conditions comprise eight positive, nine unresolved and four adverse P intervals (Table 27). Joint reporting separates evidence for replacing the baseline from evidence for retaining the adapted sample variation. Section 6 also credits Systema's analysis of systematic variation. With complete downstream scores fixed, added BRCA/GBM analyses show 43/38 positive-to-unresolved changes across removal locations, with 41/38 positive paired control-loss intervals (Table 4). These results quantify the consequences of the output comparisons and motivate Section 7's reporting guidance.