acceptodds
Under review as a conference paper at ICLR 2027

Budget-constrained Sequential Active Learning to efficiently De-censor Survival Data

Abstract

Standard supervised learners attempt to learn a model from a labeled dataset. Given a small set of labeled instances, and a pool of unlabeled instances, a budgeted learner can use its given budget to pay to acquire the labels of some unlabeled instances, which it can then use to produce a model. Here, we explore budgeted learning in the context of survival datasets, which include (right) censored instances, where we know only a lower bound on an instance’s time-to-event. Here, that learner can pay to (partially) label a censored instance – e.g., to acquire the actual time for an instance [perhaps go from (3 yr, censored) to (7.2 yr, uncensored)], or other variants [e.g., learn about one more year, so go from (3 yr, censored) to either (4 yr, censored) or perhaps (3.2 yr, uncensored)]. This serves as a model of real world data collection, where follow-up with censored patients does not always lead to uncensoring, and how much information is given to the learner model during data collection is a function of the budget and the nature of the data itself. We provide both experimental and theoretical results for how to apply state-of-the- art budgeted learning algorithms to survival data and the respective limitations that exist in doing so. For an entropy-capped batch objective, marginal greedy selection under equal costs achieves the same (1 − 1/e) approximation factor as BatchBALD. Moreover, empirical analyses on several survival tasks show that our model performs better than other potential approaches on several benchmarks. Our acquisition rule, EPIG_surv, selects a patient and a follow-up depth by the expected reduction in predictive uncertainty on a target population per unit of expected cost. Probes are acquired in batches, with the model refitted between rounds.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Survival analysis, Active learning, Budget constrained, censored data

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_75T3OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies budget-constrained acquisition of additional follow-up for right-censored survival data. EPIGsurv combines finite-window response modeling, population-targeted predictive information, and expected-cost-based patient–depth selection. The research background nevertheless needs to be completed, and several empirical conclusions require more precise support.

Critical strengths

The problem is compelling, and the survival-specific formulation and acquisition design offer meaningful methodological novelty. The evaluation across datasets and cost settings, including fresh-seed confirmation, is a strength.

Critical weaknesses

W1. Related-work positioning. The paper acknowledges its adoption of EPIG, but its relationship to broader budgeted information acquisition frameworks remains insufficiently developed. Relevant work includes Li et al., Batch Multi-Fidelity Active Learning with Budget Constraints (NeurIPS 2022), and Hübotter et al., Transductive Active Learning: Theory and Applications (NeurIPS 2024). These respectively address budgeted input–fidelity selection and target-oriented information acquisition. How does follow-up depth differ substantively from fidelity, and which modifications are required by censoring, repeated follow-up, and outcome-dependent costs?

W2. Statistical evidence and target-sample sensitivity. (a) Seeds and the closest competing baseline. Appendix E reports choosing 40 seeds for the variable-depth confirmation using a pilot paired effect size of d = 0.39, without specifying the target power or precision and corresponding calculation. Separately, the uniform-cost confirmation in Table 20 provides limited evidence of superiority over C-BALD: the AULC difference (C-BALD minus EPIGsurv) is 0.0045 [−0.0013, 0.0121] on MIMIC-IV and 0.0027 [0.0004, 0.0049] on SUPPORT, with 95% bootstrap intervals. Final-NLL differences remain unresolved on both datasets. Thus, consistent superiority over C-BALD is not established in these confirmation results. (b) Target samples. The 300 → 75 diagnostic in Table 21 shows stable initial rankings, but does not establish robustness of the complete acquisition process. What motivates the choice of 300 target covariate samples, how are they sampled, and is the target set fixed across rounds?

W3. Notation and definitions. Notation should be consistent across the text, symbol table, and algorithms. For example, K denotes both posterior sample count and total queries; Algorithm 2 uses w(G) and ∆w(x | G) after W(A) was introduced for set coverage. Please define these relationships explicitly and distinguish element weights, set values, and marginal gains. Some abbreviations are also used before being defined. Please systematically check notation, definitions, and terminology throughout the manuscript, and update the symbol table, equations, and pseudocode consistently.

Questions for authors

Q1. Please compare the response models, cost structures, and selection rules. This comparison is needed to distinguish the paper’s survival-specific innovations from inherited components. Q2. Please clarify the paired-difference variability and the power or precision criterion underlying the sample-size calculation for the variable-depth confirmation. For the fresh-seed uniform-cost comparison, please explain the practical magnitude of the gains over C-BALD and align the main conclusions with the scope of the confirmation evidence. If stronger superiority claims are retained, additional independent runs should target the relevant comparisons under a prespecified precision goal. Q3. Please extend the existing 75-versus-300 comparison through the full acquisition process in one representative setting, keeping splits, budgets, and fitting protocols unchanged. Paired AULC and final-NLL differences with uncertainty would show whether initial ranking stability translates into stable predictive performance.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 75T3

Thank you for your detailed review and your insightful specific suggestions. Due to character limits, our response is split over two messages: this one covers W1/Q1, W2b/Q3 and W3, and the second covers W2a/Q2. We have also uploaded a revised version of the paper where the changes made are highlighted in blue.

W1 / Q1. Relation to budgeted and target-oriented information acquisition

We have added and discussed both references in the revised paper, in the related work (Section 2) and in a comparison table in Appendix L.7. Hübotter et al. (2024) select data by their information about a set of target points, the same principle that EPIG and our rule use, but their queries return the label in full at the same cost. Li et al. (2022) select both inputs and fidelities under a budget, where a higher fidelity gives a more accurate label at a known cost.

Follow-up depth is different from fidelity in several ways, and each difference requires a change to the method. A probe returns one of three outcomes (an event inside the follow-up window, a censoring inside the window, or no endpoint by the end of the window), not a label at a chosen accuracy. Its cost depends on the unknown outcome, because follow-up stops early if the event happens, so it is predicted from the posterior. The same patient can be followed up again, starting from where the previous follow-up ended. And whom to follow up must be decided from observed data only, otherwise censoring becomes informative and the model is biased (Appendix K shows this with a policy that uses the true outcomes). None of the frameworks above addresses these four points together, which is why our method is more directly suited to survival follow-up.

These differences are why our method needs components that EPIG does not have (Section 4): the probabilities of the three possible follow-up outcomes; the prediction of follow-up cost; hypothetical new outcomes for the target patients as the prediction targets, instead of their partially observed records; the rule that repeated follow-up uses only observed data; and the capped batch objective. New experiments show that these components matter. Modelling the follow-up response and targeting predictions each give a significant gain (AULC on MIMIC-IV +0.0677 and +0.0285; on SUPPORT +0.0267 and +0.0067; see Table 3 in the main text). In a simulation where several probes carry the same information, greedy selection on the capped objective has 15% lower loss than selecting the top-bb probes by single-probe score (see our response to Reviewer djw1, Question 2). On MIMIC-IV, C-BALD without our follow-up response model is no better than Random, so this component matters (see our second message).

W2b / Q3. The target sample

The 300 target patients are drawn uniformly at random from the pool, using their covariates only (never their outcomes), with a fixed random seed for each run. The same targets are used in every round of a run. We chose 300 so that computing the information about the targets costs little compared with computing the follow-up response probabilities.

As you requested, we extended the 75-versus-300 comparison from the first round to the complete acquisition process, on SUPPORT with uniform costs, keeping the data splits, budgets and model fitting unchanged. Using 75 targets instead of 300 changes AULC by +0.0007 [−0.0008, +0.0022] and the final NLL by +0.0013 [−0.0018, +0.0046]. Neither difference is significant. The stability of the selected batches at the first round therefore carries over to the final predictive performance. This experiment is now in Appendix L.5 of the revised paper.

We also tested a target set drawn from a subgroup of patients (the oldest quarter by age), with the test NLL computed on the same subgroup. EPIG$_\text{surv}$ is still better than C-BALD on both MIMIC-IV (+0.0097 [+0.0066, +0.0130]) and SUPPORT (+0.0031 [+0.0005, +0.0062]).

W3. Notation and definitions

We have made the notation consistent across the text, the symbol table, the equations and the algorithms. KK now denotes only the number of posterior samples, and the total number of queries has its own symbol. In Algorithms 1 and 2, we now define the weight of an element w(e)w(e), the value of a set of sets W(A)W(A) (the total weight of the elements it covers), and the marginal gain ΔW(x∣G)=W(G∪{x})−W(G)\Delta W(x \mid G) = W(G \cup \{x\}) - W(G), and use these consistently. Every abbreviation is now defined at its first use, and the symbol table has been updated to match (Appendix A.1).

AuthorsOpenReview ↗

Response to Reviewer 75T3 Question (W2a/Q2)

W2a / Q2. Sample size, variability and the comparison with C-BALD

The number of seeds for the variable-depth confirmation (50 per dataset) was chosen before the confirmation runs, to resolve the weakest effect seen in the five-seed development pilots: a paired standardised effect of d = 0.39. "Paired" means that for each seed we take the difference between our score and the comparator's, and d is the mean of these differences divided by their standard deviation. With d = 0.39 and a two-sided paired test at α = 0.05, the power is about 0.79 with 50 seeds and 0.69 with 40 (usual normal approximation). The 95% interval has a half-width of about 0.28 standard deviations, so an effect of d = 0.39 is expected to give an interval that excludes zero, but only just. This calculation is now in Appendix L.6 of the revised paper.

The paired differences vary a lot between datasets. From the 95% intervals of the comparison with C-BALD (50 seeds), their standard deviation is about 0.018 on MIMIC-IV, 0.0014 on SUPPORT and 0.0011 on FLCHAIN (AULC units), and the observed differences (+0.0084, +0.0002, −0.0001) are about 0.47, 0.14 and −0.09 standard deviations. So 50 seeds resolve the difference on MIMIC-IV, where the effect is larger than the d = 0.39 we planned for, but not on SUPPORT and FLCHAIN, which we report as unresolved, not as equivalent. The fresh-seed uniform-cost comparison used 10 seeds. The standard deviation of its paired differences is about 0.011 on MIMIC-IV and 0.0036 on SUPPORT, and the observed differences are about 0.42 and 0.75 standard deviations. With 10 seeds the power to detect 0.4 standard deviations is only about 0.25, which is why the MIMIC-IV interval includes zero; 80% power would need about 50 seeds.

The C-BALD in our tables is a strengthened version of the published method. Published C-BALD (Hüttel et al., 2025) assumes that a query reveals a patient's full outcome, and does not model a follow-up window that stops after a limited time and may reveal nothing. To run it in our setting, we gave it our follow-up response model (Section 4.2): for each posterior draw we derived the probability of an event inside the window, a natural censoring inside the window, or no endpoint by the end of the window, conditioning on the patient having already survived to their current censoring time. Published C-BALD cannot express this. We ran it as published, on the full observed time and censoring indicator: on MIMIC-IV it is no better than Random (average error 1.5880 vs 1.5841), and clearly worse than the adapted version in our tables (1.5028; difference +0.0852 [+0.0626, +0.1123], on all 10 seeds). So the C-BALD in our tables is the published idea running on our contribution, and the comparison with it is the harder one: it measures only the value of targeting predictions instead of parameters. EPIG$_\text{surv}$ also values what is evaluated (the log-loss on the target patients), has a batch objective with a guarantee, and accounts for follow-up cost and depth, which C-BALD does not.

Real-data results against C-BALD. There is no significant loss to C-BALD in any setting, on any dataset, with respect to NLL or AULC, and EPIG$\text{surv}$ is significantly better on AULC in 11 of 14 dataset-and-setting comparisons. With uniform costs the gain is +0.0131 [+0.0057, +0.0202] on MIMIC-IV, +0.0022 [+0.0002, +0.0047] on SUPPORT and +0.0013 [+0.0007, +0.0020] on FLCHAIN. EPIG$\text{surv}$ reaches the NLL that C-BALD reaches with the full budget using a median of 51% (MIMIC-IV), 70% (FLCHAIN) and 78% (SUPPORT) of the budget, i.e. 22–49% less follow-up. On MIMIC-IV the advantage holds in every variant we ran: after removing the effect of refitting (+0.0180 [+0.0131, +0.0228]), with twice as many posterior members (+0.0090 [+0.0038, +0.0146]), and on the oldest-quarter subgroup (+0.0097 [+0.0066, +0.0130]). On new seeds it is significant on SUPPORT (+0.0027 [+0.0004, +0.0049]) and positive but not significant on MIMIC-IV (+0.0045 [−0.0013, +0.0121]). The full per-setting table, with intervals, is in Appendix L.2 (Table 25) and Section 6.1 of the revised paper, which now report the comparison with C-BALD per dataset with its interval and the budget saved, instead of just stating uniform superiority. The gain is not from finding more events: EPIG$\text{surv}$ reveals fewer events than C-BALD (63 vs 76 on FLCHAIN and 73 vs 82 on SUPPORT) and still ends with a better model. In the simulations with an exactly computed posterior (Appendix C), EPIG$\text{surv}$ has less than half of C-BALD's loss at every batch size from 1 to 20 (for example 6.1 vs 14.6 at batch size 1 and 8.1 vs 18.0 at 20), and is never worse than C-BALD in the seven simulated settings fixed in advance.

AuthorsOpenReview ↗

Comment from Authors

We thank you for the time you have put into reviewing this work. Please let us know if you have any other questions. We will keep working on the manuscript to make it more clear and incorporate any further suggestions you give us.

Reviewer_56xKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a budgeted learning method for survival prediction. It focuses on right-censored survival data and suggests ways for follow-up acquisition under budget constraints, taking into account uniform costs, random costs, and realized costs. The proposed EPIGsurv ranks actions by target-population predictive information gain from finite follow-up responses per unit of expected cost, allowing repeated probes and model refitting between batches. Experiments on three real-world datasets, various cost settings, and seven baselines show that in most paired comparisons NLL and AULC can be lower. The results are further supported by fresh-seed experiments and a second-learner experiment.

Critical strengths
  1. The formulation and the proposed method solve a practical problem: acquiring partially informative follow-up under budget constraints, with patient and depth selection guided by predictive information gain per unit of expected cost.
  2. The theoretical results provide an approximation bound for marginal greedy selection on the entropy-capped surrogate under equal costs and fixed depth,
  3. The experimental evaluation covers three real-world datasets and seven baselines, with various costs and depth settings, fresh seed validation, and a second learner.
Critical weaknesses
  1. In the experimental evaluation, the contributions of initial model fitting and adaptive depth selection remain insufficiently isolated.
  2. Some math symbols used are confusing, e.g., on page 3, b denotes the number of distinct patents, while b_j is for a event-time bin; in Step 2 of Algorithm 1: {d_i \in D | c_i < B-cost} where d_i has no obvious connection with c_i.
Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to Reviewer 56xK

Thank you for your insightful review and for the positive assessment of the formulation, the theory and the evaluation! Please see below our replies to your comments. We have also uploaded a revised version of the paper where the changes made are highlighted in blue.

Question 1(a). Isolating the contribution of the initial model fit

In our experiments, the initial model is fitted with early stopping on a validation set, and after each round it is refitted starting from the previous fit. Because each refit continues the optimisation, part of the improvement over the budget comes from the refits themselves, which is what the refit-only control in the paper measures (it receives the same refits but no new data).

To isolate this, we refit the initial model 40 times with no new data before acquisition starts, which uses up the improvement that refitting alone can give. After this, the refit-only control no longer improves (its NLL goes from 1.595 to 1.625 on MIMIC-IV and from 2.682 to 2.686 on SUPPORT), so all later improvement comes from the acquired follow-up. In this setting, EPIG$_\text{surv}$ is still clearly better than Random (AULC +0.0760 [+0.0566, +0.0973] on MIMIC-IV and +0.0205 [+0.0091, +0.0317] on SUPPORT), and its margin over C-BALD on MIMIC-IV increases from +0.0131 to +0.0180 [+0.0131, +0.0228]. The advantage therefore does not depend only on the initial fit.

We also separated the two parts of our acquisition rule. Starting from BatchBALD, we first replaced the full event time by what the follow-up can actually return (an event inside the window, a censoring inside the window, or no endpoint yet), and then replaced information about the parameters by information about the predictions for the target patients (uniform costs; paired AULC differences, where positive means the step helps):

Step 1: use the follow-up response

MIMIC-IV: +0.0677 [+0.0477, +0.0877] W

SUPPORT: +0.0267 [+0.0195, +0.0341] W

FLCHAIN: +0.0017 [-0.0009, +0.0040] T

Step 2: then target predictions instead of parameters

MIMIC-IV: +0.0285 [+0.0221, +0.0352] W

SUPPORT: +0.0067 [+0.0035, +0.0103] W

FLCHAIN: +0.0015 [+0.0006, +0.0023] W

Both parts contribute: modelling the follow-up response gives the larger gain, and targeting predictions gives a further significant gain on all three datasets.

Question 1(b). Isolating the contribution of adaptive depth

In the variable-depth setting (Appendix G), the learner chooses, for each patient, how long to follow up, from a menu of three depths. We wish to clarify that the aim of this setting was to show that EPIG$_\text{surv}$ still works on other, more complicated settings beyond those provided in the main paper (for example, when depth is also a decision), and the results show this (no resolved loss to any baseline on 63 paired comparisons over 50 fresh seeds per dataset). We did not set out to show that choosing the depth improves the results. We have made this point clearer in the revised paper and have removed the phrase "depth-menu advantage", as we now understand it may be confusing about our intentions.

To clarify, we ran our rule a second time with the same data, learner, budget, refitting schedule and division by expected cost, but with only the middle depth available. We compared the two versions on the same seeds, using the paired difference in AULC (the average error over the budget; positive means the full menu is better). On MIMIC-IV, with 50 seeds, the difference is +0.0008 [-0.0002, +0.0040], which is not significant, and the interval rules out a benefit larger than about 0.004. On SUPPORT and FLCHAIN we ran the test on 10 seeds and also found no significant difference. So, on these datasets, we could not establish a statistically significant value of the depth choice.

Question 2.

Thank you for pointing these out. In the revised paper, the event-time bins are now denoted βj\beta_j, so the letter bb denotes only the batch size. In Algorithm 1, we now state that each candidate did_i in the pool D\mathcal{D} has cost cic_i, which makes the condition defining the feasible candidates clear. We also checked the notation throughout the paper and updated the notation table (Appendix A.1) to match.

Revisions of the paper:

Thanks to your comments, we added the following to the revised paper. Question 1(a): the 40-refit control is in Appendix L.3, and the step-by-step comparison with BatchBALD is in Table 3 (Section 6.1) and Appendix Table 26. Question 1(b): the depth comparison is in Appendix L.4, and Section 6.3 now states that the variable-depth setting is not designed to show a benefit of choosing the depth. Question 2: the renamed symbols are in the notation table (Appendix A.1, Table 4) and in Algorithms 1 and 2 (Appendix A.6).

AuthorsOpenReview ↗

Comment from Authors

We thank you for the time you have put into reviewing this work. Please let us know if you have any other questions. We will keep working on the manuscript to make it more clear and incorporate any further suggestions you give us.

Thank you for answering.

Reviewer_djw1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at active learning for survival data, where some patients are censored and the learner can pay to follow them up for longer. A follow-up may reveal the event, a censoring, or nothing yet, and a patient can be followed up again. The proposed rule, EPIG$_{\rm surv}$, picks the patient/follow-up pairs that most reduce predictive uncertainty on a target population per unit of expected cost. The paper also gives a (1−1/e)(1-1/e) guarantee for a capped version of the batch objective, and reports experiments on three clinical datasets where the method beats seven baselines.

Critical strengths

The problem seems practically relevant, and the main idea makes sense to me: what you pay for is extra follow-up, not a label, so the rule should value what the follow-up will actually reveal. The experiments also look carefully done (40 paired seeds, confirmation on fresh seeds, negative results reported).

Critical weaknesses

I'm not sure the theory says much about the method that is actually used. As far as I can tell the cap rarely matters, since each term is bounded by a much smaller quantity, so the objective is basically a sum of single-patient scores and the "(1-1/e) approx factor as BatchBALD" claim in the abstract is not very meaningful. And when the cap does matter, the experiments use a simple selection rather than the greedy procedure the guarantee is about.

I'm not convinced the gains come from the proposed rule rather than from the setup. Most baselines ignore the follow-up window, which is precisely the weakness the paper points out, and against the one baseline that does use it (C-BALD) the gains are small and often not significant on new seeds. The main model also seems poorly fitted at the start: refitting alone, without any new data, accounts for a big proportion of the improvement, and the paper's own random forest starts better on two of the datasets than the main model ends. I'm not an expert in survival models, but this makes me wonder how much of the effect would remain with a well-tuned learner.

The paper is also hard to follow for a non-specialist: the main text is very dense, and many results only appear as win/tie/loss counts.

Questions for authors
  1. Why is the initial model not trained to convergence, and do the results hold if it is?

  2. How often does the cap actually matter on the real datasets?

  3. Could you compare against a version of BatchBALD that also uses the follow-up response, to show where the gain comes from?

4 comments
AuthorsOpenReview ↗

Response to Reviewer djw1

Thank you for your careful review. Please find our response to your questions and comments below. Please note that, due to character limitations, we have put our response to Question 2 in a separate comment. We have also uploaded a revised version of the paper where the changes made are highlighted in blue.

Question 1:

The initial model is fitted with early stopping on a held-out validation set, because the initial labelled set is small and training to convergence on it would overfit. After each round of follow-up, the model is refitted starting from the previous fit (a warm start), and every method uses the same refitting schedule. We warm-start because, in a check made before any acquisition run, it matched refitting from scratch within noise while being much cheaper.

To address your question, we conducted a test in which we brought the initial model as close to converged as refitting allows: before acquisition starts, we refit it 40 times with no new data. After this, the refit-only control, which receives the same refits but no new data, no longer improves (its NLL, averaged over the 10 runs, goes from 1.595 to 1.625 on MIMIC-IV and from 2.682 to 2.686 on SUPPORT between the first and last round). Any later improvement therefore comes from the follow-up that is acquired. This does not change our conclusions:

EPIG$_\text{surv}$ is still clearly better than Random (AULC +0.0760 [+0.0566, +0.0973] on MIMIC-IV and +0.0205 [+0.0091, +0.0317] on SUPPORT, both W).

Its margin over C-BALD on MIMIC-IV increases from +0.0131 to +0.0180 [+0.0131, +0.0228] (W, better on all 10 seeds).

The random survival forest does start from a better fit than the Bayesian MTLR on two datasets. Please see Appendix H for the full results: with the forest as the learner, EPIG$_\text{surv}$ again has no significant loss to any baseline on SUPPORT and FLCHAIN, and it is significantly better than most of them (3 of 7 on SUPPORT, 6 of 7 on FLCHAIN). The results are inconclusive on MIMIC-IV. The revised paper reports the 40-refit control in Appendix L.3.

Question 3:

We agree this has to be separated. Existing rules give credit for uncertainty beyond the follow-up window, which a probe can never reveal (Section 4.2), and that is a real weakness. But our advantage is not only because other methods ignore the window. To show this, we ran an ablation that starts from BatchBALD and adds one ingredient at a time, keeping the learner, data, budget and refitting schedule the same (uniform costs):

BatchBALD: information about the model parameters, scored as if a probe revealed the patient's whole event time (a probe only reveals what happens inside the follow-up window). Batches are chosen jointly by greedy selection.

BatchBALD on the follow-up response: the same algorithm, using only what a probe can actually return (an event inside the window, a censoring inside the window, or no endpoint yet). This is the version requested in your Q3.

EPIG$_\text{surv}$: also uses the follow-up response, but values a probe by what it tells the model about its predictions for the target patients, instead of about its parameters.

Mean AULC of each method (lower is better; also Table 3 in the main text of the revised paper):

Mean AULC (lower is better)

Method                  MIMIC-IV SUPPORT   FLCHAIN
Random                  1.5841    2.6487    1.1795
BatchBALD               1.5860    2.6648    1.1789
BatchBALD (response)    1.5182    2.6380    1.1772
C-BALD                  1.5028    2.6336    1.1771
EPIG_surv (ours)        1.4897    2.6313    1.1758

The results that using the follow-up response is an important step. Without it, BatchBALD is no better than Random (unresolved on MIMIC-IV and FLCHAIN, worse on SUPPORT). With it, the same algorithm clearly beats Random. Targeting predictions instead of parameters gives a further, significant gain on all three datasets, on top of a method that already uses the follow-up response.

The gain does not come from how the batch is formed. BatchBALD on the follow-up response chooses its batches jointly, which accounts for overlap between probes; EPIG$_\text{surv}$ uses simple top-bb and is still better. Our result does not rely on weaker baselines. The two methods that measure parameter information from the follow-up response (C-BALD and BatchBALD on the follow-up response) are better than every other baseline, and EPIG$_\text{surv}$ is better than both.

All methods use the same posterior, and the methods that use the follow-up response share the same response probabilities (Appendix D).

Notes on presentation: thanks to your helpful suggestions, we have made an effort to improve the paper's presentation. We have made the main text less dense and replaced most win/tie/loss counts with effect sizes, their intervals, and the budget saved. Tables of the actual values with standard errors are already in the appendix (Tables 2 and 16-17), and we will refer to them from the main text.

AuthorsOpenReview ↗

Response to Reviewer djw1 (Question 2)

Please find below our response to your Question 2:

We agree with you that the cap rarely changes the selection on our real datasets; however, the theory predicts this. A probe can tell the model at most eve_v about a target vv, where eve_v is the part of the target's uncertainty that comes from not knowing the model parameters. The cap limits a target's total credit to hvh_v, its total uncertainty, which also includes outcome noise that no probe can remove. Proposition A.3 shows that if a batch of bb probes cannot exceed the cap, i.e. b ev≤hvb\,e_v \le h_v, then the cap never changes the objective. On clinical data outcome noise is a large part of hvh_v, so this usually holds.

When the cap does not change the objective, the top-bb rule we run is not an approximation: it selects exactly the best batch for the capped objective. If no target's cap binds on the top-bb batch, the capped objective equals the plain sum of single-probe scores on that batch, and no other batch can have a larger sum. We checked this on every round of the recorded runs (uniform costs, 3 seeds, 40 rounds each). It held in 345 of 360 rounds: all 120 on SUPPORT, all 120 on FLCHAIN, and 105 of 120 on MIMIC-IV. So on most rounds the selection we ran is provably optimal for the capped objective, and the theory gives a per-round check of whether it is.

However, we believe the cap is still an important addition: namely, it matters when probes overlap, and it is what lets us handle that case with a guarantee. When several probes tell the model the same thing about the same targets, the natural batch objective (the total information the batch gives about the targets) is not submodular for prediction-targeted rules, even when probes are conditionally independent given the parameters: two probes can each be uninformative alone and informative together (Appendix A.7). Greedy selection on such an objective has no approximation guarantee. Capping each target's credit at its total uncertainty makes the objective submodular and leaves every single-probe score unchanged, which gives the (1−1/e)(1-1/e) guarantee for greedy selection. This matters in practice when probes overlap, as the following experiments show.

Simulation with redundant probes: The setting has 1,000 censored patients, all recently enrolled (no follow-up yet), in 50 subgroups of unequal size: the largest holds about 60% of the patients, the second about 15%, and most of the others only a few. Each subgroup has one unknown risk level, low (2% chance of an event in each time bin) or high (90%), equally likely at the start, shared by every patient in it. Time has 10 bins, and a probe follows up one patient for 3 bins. The 300 target patients are drawn at random from the pool, and error is measured on a separate test set of 1,000 fresh patients with the same subgroup sizes. Each method picks a batch of 3 patients, observes the results, updates, and repeats for 10 rounds (30 follow-ups in total).

Error (average over rounds / final)

Greedy on capped objective:             0.110 / 0.022
Top-b (no cap):                         0.130 / 0.028
C-BALD:                                 0.153 / 0.067
Random:                                 0.161 / 0.077

The results demonstrate that when probes are redundant, the cap changes the selection, prevents double counting, and recovers most of the achievable value. If we increase the number of duplicates, the cap becomes more and more useful. Furthermore, it is possible to set up situations with enough duplicates where top-bb performs very badly without the cap, as it continuously chooses the highest-scoring probes without accounting for redundancy.

Duplicate records on MIMIC-IV: To test a case where the cap does matter, we copied 10% of the patients 5 times each and used batches of 40 (10 seeds). Greedy selection on the capped objective is better than top-bb: AULC +0.0064 [+0.0005, +0.0136] and final NLL +0.0118 [+0.0042, +0.0212], both significant. Against C-BALD, its final NLL is better (+0.0089, significant).

Method                        AULC  Final NLL
Greedy on capped objective  1.5599     1.5079
Top-b (no cap)              1.5663     1.5196
C-BALD                      1.5673     1.5168
Random                      1.6509     1.6255

We hope this simulation and test show the experimental and theoretical utility of the cap. The revised paper will report both results (Appendix L.1).

Abstract: We agree the current sentence, which compares our guarantee with BatchBALD's, is not informative. We will replace it with: "the cap makes the prediction-targeted batch objective submodular, so greedy selection has a (1−1/e)(1-1/e) guarantee".

Finally, we understand that you have questions regarding our performance against C-BALD. For that, please refer to our response to Reviewer 75T3 (W2a/Q2).

Thank you for the careful response and the new experiments. They address my two main concerns. First, the gains are not just due to a poorly fitted initial model. Also, they are not just due to the baselines ignoring the follow-up window: when BatchBALD is given the same follow-up information as EPIG, it improves, but EPIG is still better. The gains over C-BALD remain small, so the practical impact is moderate, but my main concerns are addressed and I will raise my score to 3.

AuthorsOpenReview ↗

Response to Reviewer djw1

Thank you for engaging with our response and for updating your score. We appreciate your feedback; it improved the paper. We will be happy to address additional questions/comments that you may have. We will keep working on the manuscript to make it more clear and incorporate any further suggestions you give us.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers and the Area Chair

We thank all reviewers for their careful and constructive reviews, and for engaging with our responses during the discussion, and we thank the Area Chair for overseeing the review process. We are grateful for the reviewers' positive overall assessment of our work and that they found the problem practically relevant (djw1), with meaningful methodological novelty (75T3), and the evaluation careful (djw1, 56xK, 75T3). Their suggestions have made the paper clearer and stronger. Below we give a short overview of the discussion. All changes in the revised PDF are highlighted in blue.

Where gains come from - Details and results in Reviewers djw1 (Q1&3), 56xK (Q1(a))

Reviewers djw1 and 56xK asked whether our improvements could come from the experimental setup rather than from the acquisition rule. One possibility was that the initial model could have been trained better on the same data, so that any later refit would improve it even without new follow-up. Another was that some baselines score a follow-up as if it revealed the patient's exact event time, although follow-up lasts only a finite window and reveals only what happens within it, and that this could explain EPIG_surv's advantage. We ran two new experiments. First, we refitted the initial model on the original data until refitting alone no longer helped; our advantage remained. Second, a step-by-step comparison starting from BatchBALD showed that both ingredients of our rule contribute (scoring the outcome a follow-up actually reveals within its window, and targeting the predictions we care about rather than the model parameters), including against a BatchBALD variant that scores the same within-window outcome. Reviewer djw1 confirmed that these results address their main concerns and raised their score.

Value of entropy cap - Details and results in Reviewer djw1 (Q2)

Reviewer djw1 was concerned that the cap in our batch objective rarely matters in practice, so the accompanying theory might say little about the method. We explained that the theory predicts this: the cap only changes the selection when several probes carry the same information, and a per-round check showed that on most recorded rounds our simple selection rule is provably optimal for the capped objective. We also illustrated what the cap does when probes overlap, with a test using duplicated patient records and a simulation built to contain redundant probes. We have revised our paper thanks to these suggestions.

Comparison with C-BALD - Details and results in Reviewers djw1 (point 3) and 75T3 (W2a/Q2)

Reviewers djw1 and 75T3 commented that our gains over C-BALD, the closest baseline, are small and not always resolved on new seeds. We explained that the C-BALD in our tables already uses our follow-up response model, which makes it a strong comparator; run as originally published, it is no better than Random. The revised paper now reports this comparison per dataset, with intervals and the budget saved, and states its scope plainly: our method has no resolved loss in any setting, a resolved (statistically significant) gain in 11 of 14 comparisons. EPIG_surv reaches C-BALD's final NLL with 51–78% of the budget. Reviewer 75T3 indicated that our response resolved their questions, and Reviewer djw1 agreed their main concerns were addressed while noting that these gains are modest, which the paper now brings more light to. We emphasize two points: first, the setting is difficult: the training data are heavily censored, and a follow-up can itself end in a censoring or with no event by the end of the window, so each probe returns only partial information. In this setting, a margin that looks small on AULC can be significant (corresponds to 22–49% less follow-up to reach the same model). Second, EPIG_surv has a (1−1/e) theoretical guarantee that C-BALD does not which allows it to perform better in settings where probes overlap, as the experiments in "Value of the entropy cap" show.

Further suggestions

The reviewers also made several suggestions, all of which we adopted. We clarified the purpose of the variable-depth setting and isolated the effect of choosing the follow-up depth (Reviewer 56xK, Question 1(b)). We added the related work on budgeted multi-fidelity and transductive active learning with a comparison table (Reviewer 75T3, W1/Q1), the power calculation behind the seed count (Reviewer 75T3, W2a/Q2), and the target-set comparison over the full acquisition process (Reviewer 75T3, W2b/Q3). Finally, we resolved the notation issues raised, defined abbreviations at first use, and added worked examples and plain-language summaries to the densest parts of the main text (Reviewer 56xK, Question 2; Reviewer 75T3, W3; Reviewer djw1, presentation notes). Reviewer 75T3 confirmed they were satisfied and raised their score.

We are grateful the discussion has led to a clearer and stronger paper. We will keep working on the manuscript and incorporate any further suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.