- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies budget-constrained acquisition of additional follow-up for right-censored survival data. EPIGsurv combines finite-window response modeling, population-targeted predictive information, and expected-cost-based patient–depth selection. The research background nevertheless needs to be completed, and several empirical conclusions require more precise support.
Critical strengths
The problem is compelling, and the survival-specific formulation and acquisition design offer meaningful methodological novelty. The evaluation across datasets and cost settings, including fresh-seed confirmation, is a strength.
Critical weaknesses
W1. Related-work positioning. The paper acknowledges its adoption of EPIG, but its relationship to broader budgeted information acquisition frameworks remains insufficiently developed. Relevant work includes Li et al., Batch Multi-Fidelity Active Learning with Budget Constraints (NeurIPS 2022), and Hübotter et al., Transductive Active Learning: Theory and Applications (NeurIPS 2024). These respectively address budgeted input–fidelity selection and target-oriented information acquisition. How does follow-up depth differ substantively from fidelity, and which modifications are required by censoring, repeated follow-up, and outcome-dependent costs?
W2. Statistical evidence and target-sample sensitivity. (a) Seeds and the closest competing baseline. Appendix E reports choosing 40 seeds for the variable-depth confirmation using a pilot paired effect size of d = 0.39, without specifying the target power or precision and corresponding calculation. Separately, the uniform-cost confirmation in Table 20 provides limited evidence of superiority over C-BALD: the AULC difference (C-BALD minus EPIGsurv) is 0.0045 [−0.0013, 0.0121] on MIMIC-IV and 0.0027 [0.0004, 0.0049] on SUPPORT, with 95% bootstrap intervals. Final-NLL differences remain unresolved on both datasets. Thus, consistent superiority over C-BALD is not established in these confirmation results. (b) Target samples. The 300 → 75 diagnostic in Table 21 shows stable initial rankings, but does not establish robustness of the complete acquisition process. What motivates the choice of 300 target covariate samples, how are they sampled, and is the target set fixed across rounds?
W3. Notation and definitions. Notation should be consistent across the text, symbol table, and algorithms. For example, K denotes both posterior sample count and total queries; Algorithm 2 uses w(G) and ∆w(x | G) after W(A) was introduced for set coverage. Please define these relationships explicitly and distinguish element weights, set values, and marginal gains. Some abbreviations are also used before being defined. Please systematically check notation, definitions, and terminology throughout the manuscript, and update the symbol table, equations, and pseudocode consistently.
Questions for authors
Q1. Please compare the response models, cost structures, and selection rules. This comparison is needed to distinguish the paper’s survival-specific innovations from inherited components. Q2. Please clarify the paired-difference variability and the power or precision criterion underlying the sample-size calculation for the variable-depth confirmation. For the fresh-seed uniform-cost comparison, please explain the practical magnitude of the gains over C-BALD and align the main conclusions with the scope of the confirmation evidence. If stronger superiority claims are retained, additional independent runs should target the relevant comparisons under a prespecified precision goal. Q3. Please extend the existing 75-versus-300 comparison through the full acquisition process in one representative setting, keeping splits, budgets, and fitting protocols unchanged. Paired AULC and final-NLL differences with uncertainty would show whether initial ranking stability translates into stable predictive performance.
6 comments
Response to Reviewer 75T3 Question (W2a/Q2)
W2a / Q2. Sample size, variability and the comparison with C-BALD
The number of seeds for the variable-depth confirmation (50 per dataset) was chosen before the confirmation runs, to resolve the weakest effect seen in the five-seed development pilots: a paired standardised effect of d = 0.39. "Paired" means that for each seed we take the difference between our score and the comparator's, and d is the mean of these differences divided by their standard deviation. With d = 0.39 and a two-sided paired test at α = 0.05, the power is about 0.79 with 50 seeds and 0.69 with 40 (usual normal approximation). The 95% interval has a half-width of about 0.28 standard deviations, so an effect of d = 0.39 is expected to give an interval that excludes zero, but only just. This calculation is now in Appendix L.6 of the revised paper.
The paired differences vary a lot between datasets. From the 95% intervals of the comparison with C-BALD (50 seeds), their standard deviation is about 0.018 on MIMIC-IV, 0.0014 on SUPPORT and 0.0011 on FLCHAIN (AULC units), and the observed differences (+0.0084, +0.0002, −0.0001) are about 0.47, 0.14 and −0.09 standard deviations. So 50 seeds resolve the difference on MIMIC-IV, where the effect is larger than the d = 0.39 we planned for, but not on SUPPORT and FLCHAIN, which we report as unresolved, not as equivalent. The fresh-seed uniform-cost comparison used 10 seeds. The standard deviation of its paired differences is about 0.011 on MIMIC-IV and 0.0036 on SUPPORT, and the observed differences are about 0.42 and 0.75 standard deviations. With 10 seeds the power to detect 0.4 standard deviations is only about 0.25, which is why the MIMIC-IV interval includes zero; 80% power would need about 50 seeds.
The C-BALD in our tables is a strengthened version of the published method. Published C-BALD (Hüttel et al., 2025) assumes that a query reveals a patient's full outcome, and does not model a follow-up window that stops after a limited time and may reveal nothing. To run it in our setting, we gave it our follow-up response model (Section 4.2): for each posterior draw we derived the probability of an event inside the window, a natural censoring inside the window, or no endpoint by the end of the window, conditioning on the patient having already survived to their current censoring time. Published C-BALD cannot express this. We ran it as published, on the full observed time and censoring indicator: on MIMIC-IV it is no better than Random (average error 1.5880 vs 1.5841), and clearly worse than the adapted version in our tables (1.5028; difference +0.0852 [+0.0626, +0.1123], on all 10 seeds). So the C-BALD in our tables is the published idea running on our contribution, and the comparison with it is the harder one: it measures only the value of targeting predictions instead of parameters. EPIG$_\text{surv}$ also values what is evaluated (the log-loss on the target patients), has a batch objective with a guarantee, and accounts for follow-up cost and depth, which C-BALD does not.
Real-data results against C-BALD. There is no significant loss to C-BALD in any setting, on any dataset, with respect to NLL or AULC, and EPIG$\text{surv}$ is significantly better on AULC in 11 of 14 dataset-and-setting comparisons. With uniform costs the gain is +0.0131 [+0.0057, +0.0202] on MIMIC-IV, +0.0022 [+0.0002, +0.0047] on SUPPORT and +0.0013 [+0.0007, +0.0020] on FLCHAIN. EPIG$\text{surv}$ reaches the NLL that C-BALD reaches with the full budget using a median of 51% (MIMIC-IV), 70% (FLCHAIN) and 78% (SUPPORT) of the budget, i.e. 22–49% less follow-up. On MIMIC-IV the advantage holds in every variant we ran: after removing the effect of refitting (+0.0180 [+0.0131, +0.0228]), with twice as many posterior members (+0.0090 [+0.0038, +0.0146]), and on the oldest-quarter subgroup (+0.0097 [+0.0066, +0.0130]). On new seeds it is significant on SUPPORT (+0.0027 [+0.0004, +0.0049]) and positive but not significant on MIMIC-IV (+0.0045 [−0.0013, +0.0121]). The full per-setting table, with intervals, is in Appendix L.2 (Table 25) and Section 6.1 of the revised paper, which now report the comparison with C-BALD per dataset with its interval and the budget saved, instead of just stating uniform superiority. The gain is not from finding more events: EPIG$\text{surv}$ reveals fewer events than C-BALD (63 vs 76 on FLCHAIN and 73 vs 82 on SUPPORT) and still ends with a better model. In the simulations with an exactly computed posterior (Appendix C), EPIG$\text{surv}$ has less than half of C-BALD's loss at every batch size from 1 to 20 (for example 6.1 vs 14.6 at batch size 1 and 8.1 vs 18.0 at 20), and is never worse than C-BALD in the seven simulated settings fixed in advance.
Comment from Authors
We thank you for the time you have put into reviewing this work. Please let us know if you have any other questions. We will keep working on the manuscript to make it more clear and incorporate any further suggestions you give us.
Response to Reviewer 75T3
Thank you for your detailed review and your insightful specific suggestions. Due to character limits, our response is split over two messages: this one covers W1/Q1, W2b/Q3 and W3, and the second covers W2a/Q2. We have also uploaded a revised version of the paper where the changes made are highlighted in blue.
W1 / Q1. Relation to budgeted and target-oriented information acquisition
We have added and discussed both references in the revised paper, in the related work (Section 2) and in a comparison table in Appendix L.7. Hübotter et al. (2024) select data by their information about a set of target points, the same principle that EPIG and our rule use, but their queries return the label in full at the same cost. Li et al. (2022) select both inputs and fidelities under a budget, where a higher fidelity gives a more accurate label at a known cost.
Follow-up depth is different from fidelity in several ways, and each difference requires a change to the method. A probe returns one of three outcomes (an event inside the follow-up window, a censoring inside the window, or no endpoint by the end of the window), not a label at a chosen accuracy. Its cost depends on the unknown outcome, because follow-up stops early if the event happens, so it is predicted from the posterior. The same patient can be followed up again, starting from where the previous follow-up ended. And whom to follow up must be decided from observed data only, otherwise censoring becomes informative and the model is biased (Appendix K shows this with a policy that uses the true outcomes). None of the frameworks above addresses these four points together, which is why our method is more directly suited to survival follow-up.
These differences are why our method needs components that EPIG does not have (Section 4): the probabilities of the three possible follow-up outcomes; the prediction of follow-up cost; hypothetical new outcomes for the target patients as the prediction targets, instead of their partially observed records; the rule that repeated follow-up uses only observed data; and the capped batch objective. New experiments show that these components matter. Modelling the follow-up response and targeting predictions each give a significant gain (AULC on MIMIC-IV +0.0677 and +0.0285; on SUPPORT +0.0267 and +0.0067; see Table 3 in the main text). In a simulation where several probes carry the same information, greedy selection on the capped objective has 15% lower loss than selecting the top-b probes by single-probe score (see our response to Reviewer djw1, Question 2). On MIMIC-IV, C-BALD without our follow-up response model is no better than Random, so this component matters (see our second message).
W2b / Q3. The target sample
The 300 target patients are drawn uniformly at random from the pool, using their covariates only (never their outcomes), with a fixed random seed for each run. The same targets are used in every round of a run. We chose 300 so that computing the information about the targets costs little compared with computing the follow-up response probabilities.
As you requested, we extended the 75-versus-300 comparison from the first round to the complete acquisition process, on SUPPORT with uniform costs, keeping the data splits, budgets and model fitting unchanged. Using 75 targets instead of 300 changes AULC by +0.0007 [−0.0008, +0.0022] and the final NLL by +0.0013 [−0.0018, +0.0046]. Neither difference is significant. The stability of the selected batches at the first round therefore carries over to the final predictive performance. This experiment is now in Appendix L.5 of the revised paper.
We also tested a target set drawn from a subgroup of patients (the oldest quarter by age), with the test NLL computed on the same subgroup. EPIG$_\text{surv}$ is still better than C-BALD on both MIMIC-IV (+0.0097 [+0.0066, +0.0130]) and SUPPORT (+0.0031 [+0.0005, +0.0062]).
W3. Notation and definitions
We have made the notation consistent across the text, the symbol table, the equations and the algorithms. K now denotes only the number of posterior samples, and the total number of queries has its own symbol. In Algorithms 1 and 2, we now define the weight of an element w(e), the value of a set of sets W(A) (the total weight of the elements it covers), and the marginal gain ΔW(x∣G)=W(G∪{x})−W(G), and use these consistently. Every abbreviation is now defined at its first use, and the symbol table has been updated to match (Appendix A.1).