acceptodds
Under review as a conference paper at ICLR 2027

Learning and Uncertainty in Generalized Bayesian Discrete-Time Survival Distillation

Abstract

External survival models can improve prediction in target cohorts with few events, but the inferential consequences of borrowing from a fixed teacher remain insufficiently characterized. Existing discrete-time survival distillation methods primarily focus on point estimation; how teacher guidance affects survival-probability uncertainty requires further analysis. We develop a generalized Bayesian framework for discrete-time survival distillation that transfers interval-level conditional event probabilities from a fixed external teacher to a flexible neural student through a prediction-level Kullback–Leibler divergence, allowing teacher and student models to differ in architecture and parameterization. The resulting update combines the target survival likelihood, teacher guidance, and structural prior regularization to define a posterior over individualized survival predictions. Our local analysis quantifies how borrowing shifts fitted survival probabilities and distinguishes curvature-based posterior variance from sampling variance, which also depends on patient-level score covariance. We derive a generalized subject-deletion predictive criterion for selecting the borrowing weight. Experiments under compatible and imperfect teacher guidance show that borrowing can improve survival prediction and reduce sampling variability, whereas excessive reliance on an imperfect teacher can worsen prediction and introduce substantial bias. Real-data analyses show improved agreement with a pooled-data reference in breast cancer survival and predictive gains from borrowing across kidney-transplant populations. The framework supports flexible survival borrowing while clarifying its inferential benefits and limitations.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Survival analysis, Knowledge distillation, Generalized Bayes, Uncertainty quantification, Discrete-Time Survival Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_R6V2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper builds on DiSKD by introducing generalized posterior uncertainty analysis and proposes a generalized Bayesian framework for discrete-time survival distillation. The authors derive explicit expressions for posterior variance, sampling variance, and the risk induced by local teacher misspecification in survival distillation. They also propose a method for selecting the strength of teacher information borrowing based on leave-one-subject-out predictive scores and evaluate the proposed framework on simulated data and real data.

Critical strengths
  1. Problem formulation:The paper investigates uncertainty quantification in survival distillation, an important issue when teacher misspecification can lead to overconfident predictions.
  2. Theoretical analysis:It presents explicit expressions for posterior variance, sampling variance, and prediction risk under local teacher misspecification.
  3. Experimental evaluation: Examining predictive performance and interval coverage under different teacher conditions provides stronger evidence for the usefulness of teacher knowledge transfer and the potential for interval miscalibration.
Critical weaknesses
  1. The results primarily specialize existing theories to survival distillation, without establishing a clearly novel analytical mechanism or substantial new theoretical insight.
  2. It remains unclear to what extent the existing theoretical results explain finite-sample behavior under fixed teacher misspecification.The authors should clearly distinguish experimental findings directly supported by their theory from those that are merely consistent with its theoretical implications.
  3. The experiments show that narrower posterior intervals after teacher borrowing do not necessarily imply reliable uncertainty estimates. However, the proposed method does not by itself guarantee interval calibration. It remains unclear whether the additional uncertainty analysis provides reliable and indispensable practical benefits.
  4. Given that uncertainty analysis is a key extension of DiSKD, more direct comparisons are needed to clarify the specific contributions of the proposed approach.
Questions for authors
  1. Compared with DiSKD and existing asymptotic theory for generalized posteriors,can the authors more clearly establish the theoretical novelty of these results, beyond their specialization to survival probability estimation?
  2. What testable predictions does the theory provide for practical scenarios with fixed misspecification? How do the authors reconcile these settings?
  3. Can the proposed uncertainty analysis reliably address interval miscalibration? Could the authors directly compare the original posterior intervals and appropriate calibration baselines and report the trade-off between empirical coverage and interval width?
  4. Could the authors conduct more rigorous ablation studies to determine whether the additional uncertainty analysis provides benefits beyond those captured by point-prediction and predictive performance metrics?
5 comments
AuthorsOpenReview ↗

Response for Q1/Q2

Overview

  • Q1/W1: Explain how teacher error, student geometry and target-data noise jointly determine useful borrowing.
  • Q2/W2: Test the predicted borrowing gain and its reversal under fixed teacher discrepancy.
  • Q3/W3: Evaluate training-data calibration, then compare interval efficiency at matched coverage.
  • Q4/W4: Hold point predictions fixed to identify the contribution of posterior spread, then assess the added effect of averaging.

Q1/W1: Theoretical contribution

Our analysis identifies how teacher error, student geometry and target-data noise jointly determine useful borrowing for patient survival probabilities. Under local mismatch, the resulting risk is

\frac{v_\phi+h^2c_\phi^2\eta^2}{(1+\eta)^2},$$ where $v_\phi$ is target-only sampling variation and $hc_\phi$ is the functional effect of teacher error. Thus, the same teacher discrepancy can imply different useful borrowing ranges across patients and horizons, according to its effect through student geometry relative to target noise. The analysis also separates sampling variability from posterior spread: borrowing can lower prediction error while making residual center bias larger relative to sampling SD. This explains the calibration patterns tested in Question 3. Our patient-deletion derivation removes both likelihood and guidance contributions; its expansion identifies a likelihood-guidance covariance term omitted by ordinary likelihood-only WAIC. DiSKD supplies the fitting loss, established generalized-posterior theory supplies the covariance framework, and PSIS supplies the estimator from existing draws. Question 2 tests the borrowing-risk predictions under fixed teacher discrepancy using independently calculated population centers and variances. **Q2/W2: Fixed teacher discrepancy** **The fixed-discrepancy experiment predicts both the initial borrowing gain and the rise in error at larger weights.** Using the known simulation law and prior, we independently solve for the population MAP and compute its Hessian and patient-score covariance, then compare the resulting predictions with 100 cohorts per configuration. The 88-configuration study includes fixed and local-discrepancy controls at $n=200,500,1000$; shifts $\pm0.25$ remain fixed across sample sizes. At $n=500$ and fixed logit shift $+0.25$, predicted and observed MAP MSE are **0.000693 and 0.000728** at $\eta=0.25$, compared with observed **0.000890** without borrowing. At $\eta=3$, predicted and observed MSE rise to **0.001928 and 0.001951**. At $\eta=0.25$, predicted and observed coverage of MAP-centered curvature intervals are **94.67% and 94.62%**. The full path is reported in the revised-paper table. These calculations quantitatively connect the borrowing gain and its bias-driven reversal to finite-sample behavior under fixed teacher error.
AuthorsOpenReview ↗

Response for Q3/Q4

Q3/W3: Calibration and interval efficiency

Training-data bootstrap calibration improves the coverage-width balance at modest borrowing. In the fitted-compatible Linear experiment, 199 patient-bootstrap refits per configuration estimate MAP variability and rescale MAP-curvature intervals. Keeping the original MAP center isolates the effect of scale adjustment. All six weights use 100 cohorts; "raw" denotes the original MAP-curvature interval.

At η=0.25\eta=0.25 and 11, calibration brings coverage to 95.0% and 94.4% while shortening intervals. At η=30\eta=30, residual teacher error remains as the bootstrap scale contracts; retaining the original width gives the lower interval score, which jointly evaluates width and the distance of missed true probabilities from the interval. The center-spread analysis explains this difference: resampling estimates variability around the fitted center, while the teacher also affects that center's distance from truth.

We next compare how much interval width each method needs at a common coverage level. The bootstrap experiment evaluates calibration using target-training data. For this separate efficiency comparison, we set each method's scale using simulation truth on 80 cohorts and evaluate it on the remaining 20, rotating across all 100. Posterior intervals use sampled quantiles; curvature and sandwich intervals use their common MAP center.

At approximately 95% coverage, posterior/curvature/sandwich widths are 0.1756/0.2260/0.2346 under calibration +0.50+0.50 and 0.1663/0.2495/0.2769 under risk-magnitude shift. Posterior intervals are shorter than both geometry-based alternatives on this common coverage scale.

Q4/W4: Ablation beyond point prediction

The matched-coverage comparison evaluates the complete interval procedures. We now hold point predictions fixed to identify the contribution of posterior spread. The SharedNN ablation uses the same selected weight, prior, cohorts and MAP when replacing sandwich by posterior-curvature intervals. The posterior-mean row then adds averaging and quantile intervals. All rows use the same 100 cohorts under calibration +0.50+0.50; RMSE pools squared errors over cohorts and survival functionals.

Prediction / intervalRMSECoverage (%)WidthInterval score
Target-only posterior.0541696.79.19531.21238
Selected MAP + sandwich.0641583.98.15572.31672
Same MAP + posterior curvature.0641590.62.18565.27263
Posterior mean + quantiles.0512794.17.17022.20235

The two MAP rows have identical prediction error. Replacing sandwich spread with posterior-curvature spread improves coverage and lowers interval score, with a paired score difference of -0.0441 (95% CI [-0.0581, -0.0301]). The final row then shows the combined contribution of posterior averaging and quantile intervals: prediction error decreases further, alongside the lowest interval score among the four procedures.

Full evidence in the revised paper

  • Q1-Q2: "Independent verification of center, spread, and risk," Tables 19-20 (pp. 40-41).
  • Q3: "Same-center intervals and bootstrap calibration," Table 29 (p. 48); "Interval efficiency at matched coverage," Table 30 (p. 48).
  • Q4: "Same-center intervals and bootstrap calibration," Table 28 (p. 47).

Most of the responses make sense to me. However, in the last paragraph of the response, the predicted and observed MSE are too small to be true to me. Based on only 500 samples, why the MSE is such small? The authors need to double check the number. I have no further questions.

Reviewer_1kJqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Author extended discrete-time survival knowledge distillation (DiSKD; Deng et al., 2026) to a generalized Bayesian framework. Its goal is to quantify uncertainty in patient-specific survival probabilities when a small target cohort borrows from a fixed external teacher. The student (linear or neural, architecture-agnostic) is fit with a posterior that combines three terms: the censored discrete-time target likelihood, an η\eta-weighted Bernoulli KL toward the teacher’s interval hazards, and a structural prior. All parameters are sampled with MALA.

The theory has two parts. Under exact guidance, borrowing shrinks sampling variance at rate (1+η)−2(1+\eta)^{-2} but posterior variance only at (1+η)−1(1+\eta)^{-1}, so raw credible intervals over-cover. Under n−1/2n^{-1/2} teacher mismatch, the authors derive the bias–variance trade-off, the beneficial range of η\eta, and the MSE-optimal weight. In practice, η\eta is selected by a PSIS-estimated leave-one-out criterion that removes each patient’s likelihood and teacher contributions.

Simulations with compatible and perturbed teachers show accuracy gains and interval narrowing under good guidance, and growing bias under larger teacher error. Real-data analyses on METABRIC and SRTR kidney transplantation report improved predictive scores and narrower intervals relative to target-only inference.

Critical strengths

The author presented thier study in a clear way:

  1. The paper cleanly separates posterior spread from repeated-sampling variability, and extensive experimentation and algorithmatic support is shown. Eq. (3) makes the mechanism transparent: teacher guidance is a soft-target shift plus a (1+η)(1+\eta) curvature inflation. Propositions 3 and 5 correctly show that the generalized posterior over-disperses by a factor of (1+η)(1+\eta) under oracle guidance. The linear control (Table 14) matches the predicted 1/(1+η)1/(1+\eta) and 1/(1+η)21/(1+\eta)^2 contraction rates closely, which validates the analysis.

  2. The empirical reporting is honest, and it surfaces an important negative result and it makes the work more reliable. All candidates and numerical warnings are retained and neither tried to mask. Matched MAP and posterior-mean comparisons are reported (Table 13). Most importantly, Table 15 shows that a teacher with RF2=0.99975R^2_F = 0.99975 still yields sandwich truth coverage of only 0.383 at η=30\eta=30. That is a valuable cautionary finding for the prediction-borrowing literature.

Critical weaknesses

I found few critical weaknesses and it should be addresssed on priority.

  1. The central promise, trustworthy uncertainty for survival probabilities under external borrowing, is not delivered, and the theory covers regimes that do not arise in practice.

(a) The intervals are miscalibrated by construction, in a direction the user cannot know.

(i) With a unit learning rate on ri+ηqir_i+\eta q_i, raw intervals over-cover under exact guidance: Table 7 shows 100% coverage at η=10,30\eta=10,30. (ii) Under any fixed teacher discrepancy they under-cover, and Prop. 8 shows coverage tends to 0. (iii) Under PSIS selection (Table 11), calibration +0.10+0.10 gives 79.6% mean coverage and 61.18% at the final horizon. Late-temporal mismatch gives 83.18% at the final horizon. On METABRIC, 95% intervals contain the pooled reference for only 85.5–91.3% of subject/time pairs. (iv) The mildest shift is the worst case. PSIS borrows most there (Table 10: η≥10\eta\ge 10 in 62/100 cohorts), so the prediction-oriented selector works against coverage exactly when the mismatch is hardest to detect. (v) The acceptable coverage under larger shifts in Sim II appears to be an accident: the oracle (1+η)(1+\eta) over-dispersion offsets teacher bias. The paper’s own example in B.6 shows this (κ=1,η=3\kappa=1,\eta=3 gives risk ratio 0.625 but raw coverage 0.821). (vi) Learning-rate calibration for generalized posteriors (Syring & Martin, 2019, which is cited) is discussed but not applied.

(b) Teacher uncertainty is ignored, and the implied information exceeds what the data support. Under oracle-type guidance the posterior behaves roughly as if it had (1+η)n(1+\eta)n target-equivalent subjects.

(i) METABRIC at η=30\eta=30: about 31×77≈2,40031\times 77 \approx 2{,}400 effective subjects. The pooled data (source plus target) have 1,523 subjects, and the teacher itself was trained on 1,446. (ii) Transplant: about 31×31\times target information versus roughly 19×19\times for pooled source plus target.

So the selected posteriors may be more concentrated than a pooled-data posterior would justify, yet the paper never compares their widths against the pooled reference’s own posterior widths.

(c) The theory gives no operational guidance.

(i) The n−1/2n^{-1/2} local-mismatch regime does not describe a fixed external teacher. (ii) κϕ\kappa_\phi and η∗\eta^* are explicitly unidentified from target data. (iii) The neural students used in all real-data analyses violate the local-identifiability assumptions; 522 of 524 added MLP fits carry sampling-review flags.

(d) We observed that the only practical knob, PSIS selection, has no guarantee and is unstable.

(i) Half-draw selections disagree in 33–45/100 cohorts, and some chain disagrees in 86–90/100 (Table 3). (ii) PSIS selects η=30\eta=30 and WAIC selects η=0\eta=0 in every METABRIC partition. (iii) PSIS selections sit on the grid boundary (η=30\eta=30).

  1. The empirical gains are measured against a crippled target-only comparator, and the method does not beat simply using the teacher.

(a) The comparator prior is badly miscalibrated.

(i) Zero-centered intercepts give prior mean ES(tK)=2−K\mathbb{E}S(t_K)=2^{-K} (Eq. 10). For transplant (K=20K=20) that is a prior mean one-year survival of 9.5×10−79.5\times10^{-7}, while the observed one-year event rate is about 4.3% (83/1,935). (ii) Intercept SD 10 implies a prior interval for λ1\lambda_1 of [2×10−9, 1−3×10−9][2\times10^{-9},\,1-3\times10^{-9}] (Table 1). (iii) The target-only Bayes fit is therefore prior-dominated. Its mean one-year width is 0.646 (median 0.783; Table 22). Internal Bayes is far worse than the internal point estimator: NLL 1.756 vs. 1.479 in P1 and 1.924 vs. 1.573 in P3 (Table 19). (iv) The headline results (“89% narrower” intervals, 57–61 percentage-point width reductions, deviance 0.745→0.622) are all relative to this comparator. The authors state that an event-rate-informed paired comparison “is not part of the completed evidence”.

(b) The method does not beat the teacher on held-out data.

(i) METABRIC P2: teacher vs. Bayesian DiSKD is 1.5427 vs. 1.5515 NLL, 0.12988 vs. 0.13023 IBS, 0.6593 vs. 0.6457 CtdC^{td}. (ii) METABRIC P3: the teacher is also better on NLL, IBS and CtdC^{td}. P1 is essentially a tie. (iii) Transplant: teacher deviance is 0.6145 vs. 0.6218 at PSIS-selected η=10\eta=10, and IBS is identical (0.03274). (iv) In partition 2 the posterior mean is much worse than its own MAP on reference RMSE (0.054 → 0.090). (v) METABRIC is a random split of a single cohort, so there is no real shift. The pooled reference shares about 95% of its training data with the teacher, so “pooled-reference agreement” largely measures agreement with the teacher.

Questions for authors

Author presented thorough study however, the main contribution, calibrated uncertainty under borrowing, is not delivered for realistic (target-mismatch) teachers. The theory largely specializes known results, and the empirical gains are confounded by a mis-specified comparator prior and by missing teacher-recalibration baselines. On real data the method does not outperform the teacher alone. Moreover, clarifications whose answers could change my assessment a little bit:

  1. Please re-run target-only Bayes on transplant and METABRIC with an event-rate-centered intercept prior, and the same prior for the borrowing arm. Do the width, deviance and pooled-reference gains survive?
  2. Please add a Bayesian teacher-recalibration baseline (posterior over intercept, slope and interval offsets of the teacher’s logits) in Sim II and the real data. Does Bayesian DiSKD outperform it in accuracy or coverage?
  3. On METABRIC and transplant, how do the η=30\eta=30 / η=10\eta=10 interval widths compare with the pooled-reference posterior’s own widths? If borrowing yields narrower intervals than the pooled-data posterior, how do you justify that information content?
  4. Please report the distribution of Pareto k^\hat k (fraction above 0.7) for METABRIC (n=77n=77, 330 parameters) and transplant at the selected η\eta, and exact-refit LOO on a subset. Given that WAIC and PSIS pick opposite extremes, how reliable is the selection?
  5. Why is the posterior mean so much worse than the matched MAP in METABRIC P2 (reference RMSE 0.054 → 0.090)? Does this indicate multimodality or poor mixing in the full-network MALA?
  6. In Table 19, P1, “Internal point” has Ctd=0.4793C^{td}=0.4793, which repeats the digits of its NLL (1.4793). Is this a transcription error? If not, the concordance is below chance.
  7. In NWTCO the sensitivity analyses “followed inspection of the test data”, and concordance favors target-only at n=400n=400. How should readers weigh this against the main claims?
7 comments
AuthorsOpenReview ↗

Response for W1

Overview

  • W1: Explain the observed coverage patterns through the center–spread analysis and evaluate interval quality directly.
  • Q1–Q3: Test the comparator prior, add Bayesian teacher recalibration, and compare interval widths with the pooled-data posterior.
  • Q4: Report the requested importance-sampling diagnostics and assess their consequences through deletion refits and reselection.
  • Q5–Q6: Explain the P2 averaging result and verify the questioned comparator values.

W1: What the uncertainty analysis explains

The theory identifies when borrowing gains coexist with residual center error. We tested this relationship using independently calculated population centers and variances. At n=500n=500, fixed teacher-logit shift +0.25+0.25 and η=0.25\eta=0.25, predicted and observed MAP MSE are 0.000693 and 0.000728, compared with observed 0.000890 without borrowing. The same calculation predicts the rise in error at larger weights. This provides a quantitative check of the bias–variance trade-off under a fixed teacher discrepancy.

The mild-calibration experiment illustrates how this trade-off affects intervals. Under calibration +0.10+0.10, PSIS reduces pooled survival RMSE from 0.0542 to 0.0307 across 100 cohorts, with mean coverage 79.62%. Along the fixed-weight path, RMSE levels off near 0.029 from η=3\eta=3 to 3030, while width decreases from 0.0969 to 0.0362 and coverage from 97.43% to 16.65%. Prediction gains persist, while continued interval contraction makes the remaining center error increasingly consequential.

We evaluate the value of posterior spread while holding point predictions fixed. Under calibration +0.50+0.50, PSIS selects η=0.25\eta=0.25 in 95/100 cohorts. At the same selected MAP, posterior-curvature intervals lower the 95% interval score from 0.3167 to 0.2726 relative to sandwich. The full posterior further lowers it to 0.2023, with coverage 94.17% and width 0.1702. The interval score combines width with the distance of missed true probabilities from the interval, separating the contribution of spread from posterior averaging.

The B.6 example uses a local Gaussian setting with κ=1\kappa=1 and η=3\eta=3: 0.625 is its MSE ratio relative to no borrowing, and 0.821 is its raw coverage. It illustrates why prediction improvement and coverage improvement are distinct. The calibration +0.50+0.50 comparison above evaluates the actual fitted teacher, selected weights and interval constructions.

We also added a training-data bootstrap calibration comparison. In the fitted-compatible Linear experiment, 199 patient-bootstrap refits per configuration rescale MAP-curvature intervals while preserving the MAP center. At η=0.25\eta=0.25 and 11, calibration gives coverage of 95.0% and 94.4%, with shorter intervals. At η=30\eta=30, coverage falls to 38.5% after scaling, and retaining the original width gives the lower interval score. This directly tests the role of scale adjustment: bootstrap variability contracts with borrowing, while residual teacher-induced center error remains.

AuthorsOpenReview ↗

Response for Q1/Q2/Q3

Q1: Does borrowing still help under an event-rate-centered prior?

The METABRIC gains persist after matching the prior in both arms. Across three partitions and intercept SDs 1 and 10, borrowing lowers NLL and IBS, improves concordance and narrows intervals in all six comparisons. For intercept SD 10, IBS reductions range from 3.63% to 8.17%. Pooled-reference RMSE also decreases in all six comparisons. For example, P2 changes from 0.1350 to 0.0809 at intercept SD 10. Both arms use the same training-derived intercept center and reference temporal prior. The center stays fixed during selection. The original joint-selected fits use no temporal smoothing, so this is a separate matched-prior control.

We are completing the corresponding transplant centered-prior and pooled-reference analyses. We will add them in a follow-up response and revision.

Q2: Does a flexible student improve on Bayesian teacher recalibration?

The recalibration comparison identifies where flexible target-student fitting adds predictive value. Our target is a student that improves on target-only fitting and supplies patient-risk posteriors. Direct teacher use and recalibration provide additional benchmarks for target adaptation.

The requested recalibration baseline has a posterior over the intercept, slope and interval offsets of teacher logits. It uses the same target-training data. Across 100 paired cohorts, Bayesian DiSKD has lower mean survival RMSE under compatible guidance: 0.01512 versus 0.02333. Under late-temporal guidance, the values are 0.01920 versus 0.02333. Intervals are narrower and interval scores lower in both settings. Recalibration has lower RMSE under the two calibration shifts and risk-magnitude shift, with coverage near 94% throughout. These RMSE values average cohort-specific RMSEs.

The real-data outcomes vary by setting. In METABRIC P2, Bayesian DiSKD has lower NLL, 1.5515 versus 1.6267, and lower IBS. Recalibration is closer to the pooled reference. Recalibration improves both outcome scores in P3; P1 favors different methods on IBS and NLL. For KI-to-KP at selected η=10\eta=10, both methods have IBS about 0.0327, while recalibration has lower NLL. These comparisons locate the gains from student flexibility relative to direct teacher adaptation.

Q3: Are borrowing intervals narrower than the pooled-data posterior?

The selected METABRIC intervals remain wider than the pooled-posterior intervals. On identical patients and horizons, selected-to-pooled mean width ratios range from 1.186 to 1.677. All nine comparisons exceed one. This directly compares the uncertainty scale with the larger-data fit.

We are completing the matched transplant pooled-posterior analysis and will add its width comparison in a follow-up update. The current figure's 88.8% refers to patients with narrower intervals than target-only inference.

AuthorsOpenReview ↗

Response for Q4/Q5/Q6/Q7

Q4: How reliable is PSIS selection when scores or selected weights vary?

Selection changes have small average prediction effects in the completed checks, with greater interval sensitivity under mild shift. We examine the deletion approximation first, then the consequences of reselection.

First, 18 full-network METABRIC P2 deletion refits retain the average pairwise ordering of three configurations. For KI-to-KP, the saved PSIS-selected η=10\eta=10 fit has 1,933/1,935 (99.90%). Next, 39 independent Sim II fits retain 17 of 20 winner-runner-up orderings. Every resulting IBS change is below 0.001.

The completed draw-split check covers 1,100 original selection paths. In each Sim II setting, half-production reselection changes IBS by less than 0.000062 and RMSE by less than 0.00105 on average in absolute value. Mild calibration has larger interval effects: mean absolute coverage changes are 4.30-10.86 percentage points.

The WAIC comparison adds an independent outcome check. With the same no-temporal prior, PSIS selects η=30\eta=30 and WAIC selects zero in all three METABRIC partitions. PSIS has lower held-out IBS in each; P2 gives 0.13023 versus 0.16413. The patient-deletion derivation removes both likelihood and guidance terms. Its likelihood-guidance covariance term explains how this score differs from likelihood-only WAIC.

Q5: Why is the posterior mean worse than MAP in METABRIC P2?

At fixed η=30\eta=30 and prior, nonlinear averaging shifts P2 survival predictions away from the pooled reference. Average predicted survival changes from 0.7877 at the MAP to 0.7607 under posterior averaging. Reference RMSE rises from 0.0540 to 0.0902. The saved-draw decomposition traces this shift through hazard averaging and the joint survival products.

The posterior mean minimizes posterior expected squared loss for the patient's survival probability. This is the decision-theoretic reason for using it. Its empirical gain varies by setting. The calibration example above separates averaging from spread; under risk-magnitude shift, averaging lowers pooled RMSE from 0.0615 to 0.0474 across 100 matched cohorts. These comparisons hold the selected weight and prior fixed.

Q6: Are the P1 NLL and concordance values transcription errors?

Independent recomputation confirms both values. The comparable-pair calculation gives 19,535/40,761=0.47925719{,}535/40{,}761=0.479257 for concordance. Recomputed NLL is 1.479301. Both round to the reported entries, including the fitted comparator's below-chance concordance.

Q7: How should the NWTCO sensitivity results be interpreted?

NWTCO is an exploratory sensitivity analysis of borrowing when external risk patterns transfer imperfectly in appendix. The analysis was designed after inspecting test results. It keeps the teacher, student architecture, prior and 435-person test cohort fixed while varying target-training size. Each borrowing weight is selected using target-training outcomes.

The teacher predicts stronger separation between risk groups than the target Kaplan-Meier estimates show. At n=400n=400, target-only concordance is 0.643 versus 0.589 after PSIS-selected borrowing, averaged over three overlapping training subsets. This identifies a setting where target-only prediction performs better.

As target-training size increases, selected borrowing weights decrease and interval widths approach those of target-only inference. At the full-size endpoint, predictions are close. The case therefore illustrates how the usefulness of borrowing depends on both transferable teacher information and the amount of target data.

Evidence locations in the revised paper

  • Q4: Appendix C.7, Tables 5-6 (pp. 29-30); C.8, Tables 9-11 (pp. 31-33); E.2, Table 43 (p. 59).
  • Q5: Appendix E.2.3, Tables 46-47 (p. 61); D.6.4, Table 27 (p. 47).
  • Q6: Appendix E.2, Table 42 (p. 59).
  • Q7: Appendix E.5, Figures 14-15 (pp. 64-65).
Reviewer_j9ZhOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper considers a setting in survival analysis, where the available dataset is limited (both in number of samples, as well as the heavy censoring). In such settings, the availability of a teacher model that can predict survival probabilities (or interval hazards) of different individuals could be beneficial. However, the training data of such teacher model is most likely from a different population and could be biased. The question the paper tries to address is two-fold: (1) how to combine the predictions from the teacher model with the available data to derive more accurate survival analysis, and (2) how to perform reliable uncertainty quantification given the potential biases. For the first one, it relies on related work such as DiSKD [1] and treat the teacher labels as soft labels, weighted by a tunable parameter η\eta. Specifically, it formulates the problem via generalized Bayesian modelling, where the augmented loss is treated as the negative log-likelihood [the augmented loss is the sum of two terms: (i) cross-entropy between the model predictions and the true labels, and (ii) the KL divergence between the model predicted probabilities and the teacher probabilities]. It then derives a theory on the relationship between the variance derived from the posterior in this Bayesian formulation, as well as the sampling variance. This theory guides why the posterior spread and sampling spread can be different and also shows that fixing the variance does not fix coverage if the teacher is biased. It also uses the standard leave-one-out idea to tune the parameter η\eta. Finally, the paper provides simulated experiments along with three real data results to show how the derived intervals compare against the true coverage.

--References--

[1] Deng, Feiyang, et al. "Discrete Survival Knowledge Distillation for Competing Risks Analysis." Forty-third International Conference on Machine Learning.

Critical strengths

I couldn't find any mistake in the theoretical results and they all seem sound. The amount of simulation experiments is also a strength of the paper.

Critical weaknesses
  1. First of foremost, the writing is more or less unreadable. It’s very hard to follow and decode the contributions from what is known in the field. Unclear and rather compressed/coded terminology is used, which as far as I understand, is not very standard. For instance, in line 050, just saying “the borrowing weight η\eta” without providing any context, or in line 55/56, writing “Borrowing increases curvature without adding leading score variability” seems hard to follow. I expect the introduction section of a paper to provide the story behind the paper for someone familiar with the field. The current version does a poor job at that. Moreover, the writing seems very defensive. E.g., lines 111/112 ends with “Inference conditions on the deployed teacher; its retraining variability is not included.” It’s unclear what that means and why that’s included in the first paragraph of the method section.

  2. The paper is mostly relying on well-known results. The problem formulation on how to incorporate teacher labels follows DiSKD [1]. It then applies the well-known theory of generalized posteriors to this setting, from papers like [2] or [3] etc. This includes the results on the difference between the posterior covariance and sampling covariance in Propositions 2 and 3. So novelty is at most modest.

  3. A good chunk of the paper is about taking a Bayesian perspective, building its posterior, and then explaining why its width is wrong. It seems that applying this Bayesian paradigm with MCMC introduces more difficulty than it solves. For instance, one alternative is to fully skip the MCMC approach: fit the model and η\eta using a similar cross-validation idea as in DiSKD [1], and then use the estimated sandwich variance A^−1J^A^−1/n\hat{A}^{-1}\hat{J}\hat{A}^{-1}/n for uncertainty quantification. This avoids the cost of MCMC chains with 256,000 transitions per candidate η\eta (Appendix E.5) and the Monte Carlo noise in selecting η\eta (in Table 3). The paper seems to be already doing this in Appendix A.6, but only as a diagnostic in a simple linear model, and it is not compared on other datasets. There, the sandwich variance matches the empirical variance , whereas the raw posterior variance is about 1+η1+\eta times too large. Neither handles teacher bias (sandwich coverage 0.383 at η=30\eta=30), so it is unclear what the posterior buys beyond extra width. It is not clear what value the sophisticated approach of fitting MCMC and then analyzing how far off its posterior variance is brings for the community.

  4. No code was provided for the experiments, as far as I could tell.

--References--

[1] Deng, Feiyang, et al. "Discrete Survival Knowledge Distillation for Competing Risks Analysis." Forty-third International Conference on Machine Learning.

[2] Kleijn, Bas JK, and Aad W. Van der Vaart. "The Bernstein-von-Mises theorem under misspecification." (2012): 354-381.

[3] Miller, Jeffrey W. "Asymptotic normality, concentration, and coverage of generalized posteriors." Journal of Machine Learning Research 22.168 (2021): 1-53.

Questions for authors
  1. What's the advantage of the proposed generalized Bayesian approach with MCMC for providing credible intervals, compared to a simpler idea depicted in point (2) under weaknesses; that is to apply a similar cross-validation idea as in DiSKD [1] to estimate η\eta, and then use the dervied estimated sandwich variance A^−1J^A^−1/n\hat{A}^{-1}\hat{J} \hat{A}^{-1} /n for uncertainty quantification?

  2. Which part of the theoretical analysis is novel and which part is the application of well-known results for this specific setting?

6 comments
AuthorsOpenReview ↗

Response for Q1

Overview

  • Q1/W3: Posterior spread, averaging, and matched-coverage efficiency address the comparison with MAP+sandwich; saved draws also support predictive selection.
  • Q2/W2: We distinguish our survival-posterior construction, borrowing-risk analysis, and patient-deletion derivation from their established foundations.

Q1/W3: Posterior intervals and averaging Posterior intervals give a better coverage–width balance under accurate guidance. To show this, we first evaluate whether the additional posterior spread improves the balance between interval width and missed true probabilities.For a 95% interval [l,u][l,u] and true survival probability ϕ0\phi_0, we use the interval score

\qquad (a)_+=\max(a,0).$$ This adds interval width to a penalty for the distance of any missed true probability from the nearest endpoint. Lower scores favor narrower intervals with fewer and smaller misses. In Simulation I’s fitted-compatible Linear control at $\eta=30$, evaluated over 100 cohorts, posterior intervals have approximately $100\%$ coverage versus $38.33\%$ for MAP+sandwich, and a lower interval score: $0.0191$ versus $0.0538$. Direct empirical-SD intervals give similar coverage to sandwich, $39.18\%$. Together, these comparisons show that residual center error remains relevant even when sampling variability is accurately estimated; retaining posterior-scale width reduces the missed-truth penalty enough to offset its additional width. **The extended borrowing path shows how this balance changes as intervals contract.** In a separate continuation using the same fitted-compatible teacher and three paired cohorts, posterior coverage reaches $92.89\%$ at $\eta=100$, with mean width $0.01060$. Increasing $\eta$ to $150$ produces only a small RMSE improvement, while coverage falls to $83.79\%$ and the interval score increases from $0.01330$ to $0.01648$. This path connects the interval results to the center–spread analysis: once prediction gains level off, further concentration makes the remaining center error increasingly consequential. **With biased guidance, posterior averaging also improves survival predictions.** In Simulation II's full-network SharedNN comparison, posterior averaging reduces RMSE from $0.06415$ to $0.05127$ under calibration $+0.50$ and from $0.06147$ to $0.04742$ under risk-magnitude shift. Posterior coverage is $94.17\%$ and $94.84\%$, compared with $83.98\%$ and $82.70\%$ for MAP+sandwich. These posterior intervals come directly from joint draws of all network parameters; the sandwich is computed separately from the full-network curvature and patient scores. Posterior averaging improves prediction accuracy, while the resulting intervals bring coverage closer to the nominal 95% level. **The posterior intervals are also narrower at matched coverage in these neural comparisons.** In a simulation benchmark, interval scales are learned using truth on 80 cohorts and evaluated on the other 20, rotating across all 100 cohorts. At approximately $95\%$ coverage, posterior-quantile widths are $0.1756$ versus $0.2346$ for MAP+sandwich under calibration $+0.50$, and $0.1663$ versus $0.2769$ under risk-magnitude shift. At comparable coverage, posterior intervals are approximately 25% and 40% narrower, respectively. **Matched coverage: MAP+sandwich $\to$ posterior intervals.** | Teacher | Coverage (%) | Mean width | | --- | ---: | ---: | | Calibration $+0.50$ | $94.95\to94.97$ | $.2346\to.1756$ | | Risk magnitude | $94.91\to94.93$ | $.2769\to.1663$ | **PSIS reuses these draws to select borrowing strength.** In METABRIC P2, fitting 18 full-data posteriors (six weights under three priors) took $2.43$ CPUh; PSIS scoring from their saved likelihood/guidance arrays took $8.00$ seconds. Three-fold full-network Bayesian CV required 54 additional fold fits, taking $7.50$ CPUh. Within each of the two weak-prior settings, both methods select $\eta=30$ and use the same full-data posterior for evaluation. The draws used for uncertainty therefore also make predictive selection inexpensive.
AuthorsOpenReview ↗

Response for Q2

Q2/W2: Theoretical contribution

Our contribution is patient-specific survival inference after borrowing external predictions. In small censored cohorts, external survival models can supply information even when their training records are unavailable. We extend DiSKD's point-fitting objective to joint posterior inference for patient survival probabilities. KL guidance supplies fractional event and non-event contributions within observed risk intervals, forming a teacher-induced pseudo-prior together with the base prior. The Gaussian component regularizes student parameters, while optional temporal smoothing links adjacent hazard logits. This separates external information from structural regularization and yields averaged survival curves with probability intervals.

The new analysis quantifies survival-probability error under teacher discrepancy and identifies beneficial borrowing. We trace how teacher error changes the survival probability of a given patient at a given horizon, and compare that change with the reduction in repeated-sampling variability. Under local teacher mismatch, this yields an explicit functional MSE and a range of borrowing weights that improve on target-only estimation. The analysis connects estimation gains to interval behavior, explaining why residual center error matters even when sampling variability is accurately estimated.

Our selection derivation accounts for teacher guidance when leaving out a patient. The score evaluates the held-out target outcome after removing that patient's likelihood and teacher-loss contributions. Its expansion identifies a likelihood–guidance covariance term omitted by ordinary likelihood-only WAIC. Refitting CV can also remove both contributions. PSIS estimates the deletion score from existing joint posterior draws, so the draws used for survival uncertainty also support weight selection without additional fold fits.

We build on the DiSKD loss, generalized-posterior covariance theory, and PSIS. DiSKD supplies the likelihood and guidance objective [1], while established covariance theory [2,3] provides the foundation for our asymptotic analysis. Gaussian shrinkage and temporal smoothing are established prior components, and PSIS supplies the importance-sampling estimator. The new contributions are the survival-specific posterior construction, the borrowing-risk analysis, and the patient-deletion calculation for this combined loss.

W1: Presentation

We will introduce the survival problem in plain language. A small, censored target cohort can borrow external teacher predictions through the weight η\eta. We will explain this setting before presenting the variance results. The fixed-teacher clarification will accompany the teacher description: the same supplied teacher is used across target datasets. This order connects the practical problem, the model, and its uncertainty analysis.

W4: Code availability

The implementation is now available anonymously. The code repository includes full-network posterior inference, patient-level PSIS selection, simulation code, and usage instructions.

References

[1] Deng, Feiyang, et al. “Discrete Survival Knowledge Distillation for Competing Risks Analysis.” Forty-third International Conference on Machine Learning. [2] Kleijn, Bas JK, and Aad W. Van der Vaart. “The Bernstein-von-Mises theorem under misspecification.” (2012): 354–381. [3] Miller, Jeffrey W. “Asymptotic normality, concentration, and coverage of generalized posteriors.” Journal of Machine Learning Research 22.168 (2021): 1–53.

Full evidence in the revised paper

  • Q1/W3, intervals: “Spread and coverage under fitted guidance,” Tables 23–25, 27, and 30.
  • Q1/W3, computation: “Full-network predictive deletion and computational reuse,” Table 8.

I thank the authors for the response and providing a link to the code. Unfortunately, the response doesn't fully address my main concern (Q1/W3) though. The paper’s own Tables 24–25 show that in the linear setting MAP + posterior curvature (no MCMC) matches the MCMC posterior in coverage, width and interval score, and the paper attributes the coverage gain to “posterior-scale width”. This width exceeds the sampling spread by roughly 1+η\sqrt{1+\eta} because of the loss scaling. Consistent with this, posterior coverage is 79.679.6 \\% under calibration +0.10+0.10 (Table 27), 13.513.5\\% under model mismatch (vs 21.121.1\\% for the sandwich), and 83.883.8\\% at η=150\eta=150. At matched coverage the posterior offers no width advantage in the linear model (Table 31).

Moreover, on Q2/W2, the response suggests that the theoretical components are adaptations of DiSKD and generalized-posterior covariance results. Therefore, I maintain my initial score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.