acceptodds
Under review as a conference paper at ICLR 2027

Auditing Forecast Outputs Under Cohort Shift: Residual Learning Retains Conversion-Ranking Information Where Predicted Change Does Not

Abstract

A probabilistic forecaster produces several scores for each subject: its predicted change, its predicted residual spread, and its variability across model fits. These scores need not remain equally informative when the model is applied to a new cohort. We study this question in Alzheimer's disease (AD), where cortical thinning tracks disease progression. Here, we train a probabilistic forecaster on longitudinal Magnetic Resonance Imaging (MRI) data from the Alzheimer's Disease Neuroimaging Initiative (ADNI) study to predict future regional brain cortical thickness, without using future-conversion diagnoses. We then freeze it and evaluate, on the Australian Imaging, Biomarker and Lifestyle (AIBL) data, how well each of three scores identifies subjects whose clinical diagnosis will worsen within 24 months (converters): predicted-change magnitude \(s_\mu\), learned residual scale \(s_\sigma\), and model variability \(s_PW\). As an in-domain reference, we evaluate a separate volumetric forecasting analysis within ADNI data, without cohort shift: on 127 held-out subjects (26 converters), its predicted change is informative (ROC-AUC ), residual scale is weaker (), and model variability is near chance (). In contrast, when the cortical-thickness forecaster is transferred from ADNI to AIBL 139 subjects; 14 converters), residual scale identifies converters with ROC-AUC , predicted change performs near chance (), and model-variability scores do not exceed chance (–). The advantage of residual scale over predicted change on AIBL is consistent across training seeds and leave-one-converter-out checks; its 95% interval includes zero at 24 months but excludes zero when the outcome window is extended to 36 and 54 months. We explain these differences theoretically: predicted change transfers only if its relationship to disease severity is preserved across cohorts; residual scale transfers if its distribution given severity is stable within each outcome class; and model variability reflects parameter sensitivity rather than disease state. These results argue for evaluating each forecast output separately under cohort shift; they do not show that uncertainty is generally superior, nor do they provide a calibrated clinical risk model.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Uncertainty quantification, Distribution shift, Transfer learning, Medical imaging, Cohort-transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5DYdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies multiple types of statistical estimators towards their usefulness in Alzheimer's disease prediction. The results don't reveal a clear pattern, sometimes performing near chance, sometimes with significant statistical correlation to AD conversion.

Critical strengths

None

Critical weaknesses
  1. I found the paper in a draft/messy state, with an incomplete Fig 1, that is placed before the abstract.
  2. The results do highlight anything new or interesting, and there are no novel methods introduced as far as I could tell.
  3. It was hard to follow the logic and narrative in the paper. Needs a complete rewrite in my opinion.
Questions for authors

None

Reviewer_z4FxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks which outputs of a probabilistic longitudinal forecaster remain useful for ranking downstream clinical risk when transferred across cohorts. Specifically, the authors train a graph network on ADNI cortical thickness trajectories, without using future clinical conversion labels, and extract three scalar scores per subject: predicted change magnitude (s_mu), learned heteroscedastic residual scale (s_sigma), and parameter/ensemble variability (s_PW). When evaluated zero-shot on AIBL to rank 24-month AD conversion, s_mu drops near chance while s_sigma retains discriminative power, even though s_sigma is uncorrelated with actual forecasting errors on AIBL. By contrast, in an in-domain volumetric experiment within ADNI, the ordering reverses (s_mu beats s_sigma), and s_PW stays near chance throughout. The authors also provide simple analytical models in Section 5 to illustrate how cohort shift can cancel the severity slope of s_mu while leaving s_sigma stable if its class-conditional distribution is preserved.

Critical strengths
  1. Interesting decoupling between forecast error, uncertainty calibration, and clinical ranking: the central empirical finding is genuinely insightful. Practitioners often assume that if a model's point forecast degrades under shift (or if its predicted variance is poorly calibrated against actual residuals), its uncertainty output is useless. Showing that the heteroscedastic scale s_sigma has near-zero correlation with actual forecast error on AIBL, yet still ranks future clinical converters reasonably well (and consistently across longer follow-up windows and ADNI phase shifts), is a valuable contribution to both medical ML and uncertainty quantification.

  2. Nuanced framing and statistical transparency: rather than overselling "aleatoric uncertainty" as universally better than point predictions, the authors carefully show that the ranking depends on the setup (e.g., s_mu beats s_sigma in the within-ADNI volumetric control, and both perform equally well in ADNI-1 to ADNI-GO/2/3 transfer). I also strongly appreciate the statistical honesty throughout the paper: the authors explicitly highlight where sample sizes are small, run exact permutation and bootstrap coverage checks, and include constant-score controls when s_mu drops below chance.

Critical weaknesses
  1. Potential confounding by baseline diagnosis and the unsigned definition of s_mu: the forecaster takes baseline clinical diagnosis (CN vs MCI) as an input covariate. Because MCI subjects convert at a much higher rate than CN subjects, any score that mostly separates baseline MCI from baseline CN will achieve a solid global AUC across cohorts. In Table 2, a logistic model trained only on baseline clinical covariates gets 0.701 AUC on AIBL, slightly above s_sigma (0.695), and in the ADNI volumetric control (Section 4.4), s_sigma drops to chance (0.55) when evaluated strictly within baseline MCI subjects. Without seeing the unpooled within-CN and within-MCI AUCs on AIBL, it is hard to tell how much s_sigma goes beyond baseline stage separation. Furthermore, s_mu is defined as unsigned RMS change across regions rather than signed cortical thinning. Since AIBL inputs are standardized with ADNI statistics, a systematic scanner offset combined with regression to the mean could cause predicted "thickening" in some subjects, scrambling the unsigned magnitude ranking and explaining why s_mu goes below chance at 36 and 54 months.

  2. Low power on the main external cohort and messy editing in the appendix: the primary 24-month AIBL test set only has 14 converters, so the headline gap between s_sigma and s_mu is not statistically significant, and the OASIS-3 probe is similarly underpowered (7 to 14 converters, all CIs crossing zero). While the within-ADNI phase shift in Appendix H.7 (113 converters) is well powered, it is buried in the supplement. In addition, the appendix needs an editing pass: Appendix J duplicates several subsections and tables from Appendices D, H, and I (e.g., Table 16 and Table 24 are identical), and there are unexplained discrepancies in reported numbers across sections (such as the inverse AIBL-to-ADNI transfer results in H.4 vs H.9 vs J.5).

Questions for authors
  1. Signed thinning vs unsigned change for s_mu: what happens on AIBL (at 24, 36, and 54 months) if s_mu is computed as signed mean cortical thinning rather than unsigned RMS change? Does it still collapse or drop below 0.5?

  2. Role of baseline diagnosis on AIBL: Can you report the unpooled AUCs of s_sigma and s_mu on AIBL separately within baseline CN subjects (n=120) and baseline MCI subjects (n=19) across the 24/36/54m windows (without pooling with ADNI validation subjects as in Table 22)? Also, does s_sigma still transfer if baseline diagnosis is removed from the input covariates c_i?

  3. Minor suggestions: I strongly suggest moving Table 14 (within-ADNI phase shift with 113 converters) into the main text to strengthen Section 4.5, removing the duplicated material in the Appendix, and adding missing citations for Deep Evidential Regression (Amini et al., 2020) and heteroscedastic NLL training issues (Seitzer et al., 2022).

Reviewer_VuTGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors train a model on ADNI that predicts how a patient's brain cortex will change over time, then freeze it and apply it to a second cohort, AIBL. They use three outputs of the model to rank patients by their risk of getting worse within 24 months: the size of the predicted change, the model's own uncertainty, and the disagreement between model versions. Their main claim is that, after moving to the new cohort, the uncertainty output still identifies patients at risk, while the predicted change no longer does. They add a simple theoretical analysis of when each output should keep working.

Critical strengths

The question is useful in practice: when a model is moved to a new hospital or cohort, which of its outputs can still be trusted? The paper keeps model training separate from clinical evaluation, and it is open about its own statistical limits.

Critical weaknesses
  1. The main result is not statistically significant. At the primary 24-month point, AIBL has only 14 patients who got worse, and the confidence interval for the gap between the two outputs includes zero. The title still states the result as a finding. The gap becomes significant only after the follow-up window is extended to 36 and 54 months. Those extensions reuse the same patients and the same predictions, so they do not independently confirm the result.
  2. The paper's own numbers do not support the claim that the cohort change causes the gap. On ADNI, with no cohort change at all, the same model's uncertainty output already beats its predicted change by a clear margin, 0.767 vs. 0.661. The reversed ordering the paper uses as a contrast comes from a different model trained on different inputs. On AIBL, simple clinical information such as diagnosis at entry, age and genotype does slightly better than the uncertainty output. Among patients who already had mild cognitive impairment at entry, the uncertainty output is no better than chance. So it may mostly reflect how sick a patient already is, which the model receives as input, rather than predicting who will get worse.
Questions for authors
  1. Were the 36- and 54-month windows chosen before seeing the results? Why does predicted change become worse than random over longer windows?
  2. On ADNI, without any cohort change, the uncertainty output is already better. What shows that the AIBL result is caused by the cohort change?
  3. Does the uncertainty output add anything beyond the patient's diagnosis at entry? Please report results separately for each diagnosis group.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.