- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper builds on DiSKD by introducing generalized posterior uncertainty analysis and proposes a generalized Bayesian framework for discrete-time survival distillation. The authors derive explicit expressions for posterior variance, sampling variance, and the risk induced by local teacher misspecification in survival distillation. They also propose a method for selecting the strength of teacher information borrowing based on leave-one-subject-out predictive scores and evaluate the proposed framework on simulated data and real data.
Critical strengths
- Problem formulation:The paper investigates uncertainty quantification in survival distillation, an important issue when teacher misspecification can lead to overconfident predictions.
- Theoretical analysis:It presents explicit expressions for posterior variance, sampling variance, and prediction risk under local teacher misspecification.
- Experimental evaluation: Examining predictive performance and interval coverage under different teacher conditions provides stronger evidence for the usefulness of teacher knowledge transfer and the potential for interval miscalibration.
Critical weaknesses
- The results primarily specialize existing theories to survival distillation, without establishing a clearly novel analytical mechanism or substantial new theoretical insight.
- It remains unclear to what extent the existing theoretical results explain finite-sample behavior under fixed teacher misspecification.The authors should clearly distinguish experimental findings directly supported by their theory from those that are merely consistent with its theoretical implications.
- The experiments show that narrower posterior intervals after teacher borrowing do not necessarily imply reliable uncertainty estimates. However, the proposed method does not by itself guarantee interval calibration. It remains unclear whether the additional uncertainty analysis provides reliable and indispensable practical benefits.
- Given that uncertainty analysis is a key extension of DiSKD, more direct comparisons are needed to clarify the specific contributions of the proposed approach.
Questions for authors
- Compared with DiSKD and existing asymptotic theory for generalized posteriors,can the authors more clearly establish the theoretical novelty of these results, beyond their specialization to survival probability estimation?
- What testable predictions does the theory provide for practical scenarios with fixed misspecification? How do the authors reconcile these settings?
- Can the proposed uncertainty analysis reliably address interval miscalibration? Could the authors directly compare the original posterior intervals and appropriate calibration baselines and report the trade-off between empirical coverage and interval width?
- Could the authors conduct more rigorous ablation studies to determine whether the additional uncertainty analysis provides benefits beyond those captured by point-prediction and predictive performance metrics?
5 comments
Response for Q3/Q4
Q3/W3: Calibration and interval efficiency
Training-data bootstrap calibration improves the coverage-width balance at modest borrowing. In the fitted-compatible Linear experiment, 199 patient-bootstrap refits per configuration estimate MAP variability and rescale MAP-curvature intervals. Keeping the original MAP center isolates the effect of scale adjustment. All six weights use 100 cohorts; "raw" denotes the original MAP-curvature interval.
At and , calibration brings coverage to 95.0% and 94.4% while shortening intervals. At , residual teacher error remains as the bootstrap scale contracts; retaining the original width gives the lower interval score, which jointly evaluates width and the distance of missed true probabilities from the interval. The center-spread analysis explains this difference: resampling estimates variability around the fitted center, while the teacher also affects that center's distance from truth.
We next compare how much interval width each method needs at a common coverage level. The bootstrap experiment evaluates calibration using target-training data. For this separate efficiency comparison, we set each method's scale using simulation truth on 80 cohorts and evaluate it on the remaining 20, rotating across all 100. Posterior intervals use sampled quantiles; curvature and sandwich intervals use their common MAP center.
At approximately 95% coverage, posterior/curvature/sandwich widths are 0.1756/0.2260/0.2346 under calibration and 0.1663/0.2495/0.2769 under risk-magnitude shift. Posterior intervals are shorter than both geometry-based alternatives on this common coverage scale.
Q4/W4: Ablation beyond point prediction
The matched-coverage comparison evaluates the complete interval procedures. We now hold point predictions fixed to identify the contribution of posterior spread. The SharedNN ablation uses the same selected weight, prior, cohorts and MAP when replacing sandwich by posterior-curvature intervals. The posterior-mean row then adds averaging and quantile intervals. All rows use the same 100 cohorts under calibration ; RMSE pools squared errors over cohorts and survival functionals.
| Prediction / interval | RMSE | Coverage (%) | Width | Interval score |
|---|---|---|---|---|
| Target-only posterior | .05416 | 96.79 | .19531 | .21238 |
| Selected MAP + sandwich | .06415 | 83.98 | .15572 | .31672 |
| Same MAP + posterior curvature | .06415 | 90.62 | .18565 | .27263 |
| Posterior mean + quantiles | .05127 | 94.17 | .17022 | .20235 |
The two MAP rows have identical prediction error. Replacing sandwich spread with posterior-curvature spread improves coverage and lowers interval score, with a paired score difference of -0.0441 (95% CI [-0.0581, -0.0301]). The final row then shows the combined contribution of posterior averaging and quantile intervals: prediction error decreases further, alongside the lowest interval score among the four procedures.
Full evidence in the revised paper
- Q1-Q2: "Independent verification of center, spread, and risk," Tables 19-20 (pp. 40-41).
- Q3: "Same-center intervals and bootstrap calibration," Table 29 (p. 48); "Interval efficiency at matched coverage," Table 30 (p. 48).
- Q4: "Same-center intervals and bootstrap calibration," Table 28 (p. 47).
Most of the responses make sense to me. However, in the last paragraph of the response, the predicted and observed MSE are too small to be true to me. Based on only 500 samples, why the MSE is such small? The authors need to double check the number. I have no further questions.
Response for Q1/Q2
Overview
Q1/W1: Theoretical contribution
Our analysis identifies how teacher error, student geometry and target-data noise jointly determine useful borrowing for patient survival probabilities. Under local mismatch, the resulting risk is
\frac{v_\phi+h^2c_\phi^2\eta^2}{(1+\eta)^2},$$ where $v_\phi$ is target-only sampling variation and $hc_\phi$ is the functional effect of teacher error. Thus, the same teacher discrepancy can imply different useful borrowing ranges across patients and horizons, according to its effect through student geometry relative to target noise. The analysis also separates sampling variability from posterior spread: borrowing can lower prediction error while making residual center bias larger relative to sampling SD. This explains the calibration patterns tested in Question 3. Our patient-deletion derivation removes both likelihood and guidance contributions; its expansion identifies a likelihood-guidance covariance term omitted by ordinary likelihood-only WAIC. DiSKD supplies the fitting loss, established generalized-posterior theory supplies the covariance framework, and PSIS supplies the estimator from existing draws. Question 2 tests the borrowing-risk predictions under fixed teacher discrepancy using independently calculated population centers and variances. **Q2/W2: Fixed teacher discrepancy** **The fixed-discrepancy experiment predicts both the initial borrowing gain and the rise in error at larger weights.** Using the known simulation law and prior, we independently solve for the population MAP and compute its Hessian and patient-score covariance, then compare the resulting predictions with 100 cohorts per configuration. The 88-configuration study includes fixed and local-discrepancy controls at $n=200,500,1000$; shifts $\pm0.25$ remain fixed across sample sizes. At $n=500$ and fixed logit shift $+0.25$, predicted and observed MAP MSE are **0.000693 and 0.000728** at $\eta=0.25$, compared with observed **0.000890** without borrowing. At $\eta=3$, predicted and observed MSE rise to **0.001928 and 0.001951**. At $\eta=0.25$, predicted and observed coverage of MAP-centered curvature intervals are **94.67% and 94.62%**. The full path is reported in the revised-paper table. These calculations quantitatively connect the borrowing gain and its bias-driven reversal to finite-sample behavior under fixed teacher error.