acceptodds
Withdrawn from ICLR 2027

Drift-Conditioned Value of Information for Costly Multi-Label Streams with Delayed Labels

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Cost-aware prediction in a multi-label stream must decide which measurement to purchase before delayed labels reveal which risks have changed. Aggregate stream state is insufficient when drift is label local and candidate features have label-specific utility. We formulate drift-conditioned value of information: a delayed residual process estimates a label-local drift posterior, and each feature is scored by its expected reduction in class-balanced decision risk per unit cost, weighted by that posterior. The resulting policy reallocates budget when and where drift aligns with feature-specific risk reduction. On the chronological MultiEURLEX stream, DRIFTBUDGET achieves 70.4 macro-F1 and 58.0 tail recall at 44.5 normalized cost, compared with 66.1, 50.2, and 46.5 for DiFA, while reducing recovery from 17.9 to 10.8 steps. Posterior, global-state, delayed-credit, and delay controls at comparable cost support label-local drift conditioning as the mechanism behind the changed acquisition policy.

Primary area:
learning on time series and dynamical systems
Keywords:
Concept Drift, Active Feature Acquisition, Delayed Labels, Multi-Label Learning, Value of Information, Cost-Sensitive Learning, Online Learning, Temporal Generalization, Budgeted Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CE6ROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies active feature acquisition for multi-label streams with delayed supervision and label-local drift. DriftBudget combines a posterior estimated from arrived label residuals with the expected label-specific reduction in class-balanced predictive risk, normalizes the resulting score by feature cost, and greedily acquires feasible features. Delayed labels subsequently update the drift state and a timestamped feature-value model. The reported empirical evidence consists of a chronological MultiEURLEX comparison and several aggregate posterior, global-state, delayed-credit, and delay controls; a broader synthetic and multi-dataset protocol is described but its principal mechanism endpoints are not reported.

Critical strengths

The problem is general and well-motivated: delayed feedback and heterogeneous label drift can make aggregate uncertainty an inadequate guide to purchasing measurements, particularly for rare labels.

Critical weaknesses
  1. This paper is not a research paper, but a technical report, and the central method is not specified at an implementable level. Eq. (2) defines a posterior but Section 4.1 gives neither a transition model nor a residual likelihood, prior, propagation rule, or concrete calibration procedure. Sections 4.2-4.3 likewise do not specify the conditional feature distribution, predictive-risk estimator, or feature-value model update. The loss, label weights, shrinkage, beta, stopping threshold, and predictor architecture are not instantiated.

  2. The evidence required to establish the claimed interaction is largely missing. Table 1 specifies heterogeneous/global drift comparisons, predicted-versus-realized value rank correlation, and misspecification/support tests; Sections 6 and 8 also name stationary equivalence and posterior-localization endpoints. None is quantitatively reported.

  3. The core score in Eq. (4) can undermine Eq. (1)’s risk–cost objective even with perfect feature-value estimates. With equal costs and label weights, beta = 0.1, and budget for one feature, risk reductions of 0.20 and 0.10 become scores proportional to 0.02 and 0.10 for drift posteriors 0 and 1, respectively. The policy selects the smaller benefit at identical cost. Please derive this multiplier from a decision model and explain why Eq. (3)’s current risk estimates need additional drift weighting.

Questions for authors

see weaknesses

Reviewer_W3h6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies feature acquisition in multi-label streams where measurements have a cost and labels arrive with a delay. The proposed method, DriftBudget, combines a label-specific drift posterior with the expected reduction in prediction risk per unit cost. The intuition is that, after a change, the acquisition budget should favour features that help the affected labels. Results on chronological MultiEURLEX show improvements in macro-F1, tail recall, and recovery time over the reported baselines.

Critical strengths

I find the motivation convincing. In a multi-label problem, changes affecting a small group of labels can easily be hidden by aggregate performance. Connecting label-local drift evidence to feature acquisition is therefore a useful direction, particularly when feedback is delayed and the budget is limited.

The reported gains are encouraging. DriftBudget improves macro-F1 from 66.1 to 70.4 over DiFA while using slightly less acquisition cost. The global-state, posterior-removal, and delayed-credit ablations are also relevant to understanding the method. I appreciate that the paper explicitly acknowledges the missing analyses and avoids claiming broad generalisation from a single dataset.

Critical weaknesses

My main concern is that the method is described more as a framework than as a fully specified algorithm. The paper does not explain the residual calculation, state-space transition and observation models, conditional feature model, or risk calibration in enough detail. The treatment of rare labels and the online update procedure also remain unclear. These choices could substantially affect the results, so they need to be documented even if they are not claimed as separate contributions.

The connection between the objective and the acquisition score needs more explanation. Equation 1 defines a risk objective with fixed label weights, while Equation 4 introduces additional drift-dependent weighting. I can see the intuition, but it is unclear why this particular weighting is appropriate for the stated objective. If the risk-reduction estimate already reflects the current distribution, could the drift factor overemphasise some labels? A comparison using the same risk-based acquisition model with and without drift weighting would help resolve this.

The experimental setup is also difficult to assess. For MultiEURLEX, the actual feature groups being purchased, their costs, the imposed delay distribution, chronological splits, and predictor configuration are not specified. Recovery time and tail-label selection need clear definitions. Although the protocol mentions at least five seeds, the results contain no standard deviations or confidence intervals, making it difficult to judge the stability of the gains.

More importantly, the experiments do not yet establish the central mechanism. The paper identifies value-ranking correlation, stationary/global equivalence, and misspecification tests as important, but does not report them. The synthetic study is similarly described without results. The existing ablations suggest that label-local information helps, but they do not fully demonstrate that the gains come from accurately identifying which features reduce risk for the labels undergoing drift.

Questions for authors

How is delayed credit assigned to individual acquisitions? Receiving a label allows prediction losses to be evaluated, but the procedure for estimating each feature’s marginal contribution is unclear. Are predictions stored before and after every acquisition, and how are feature interactions handled?

I would also like to understand why DriftBudget outperforms the drift-aware full-feature predictor. This is possible, but deserves investigation. Are predictor capacity, training data, and adaptation schedules matched? Finally, what is the computational overhead of repeatedly estimating conditional feature values during acquisition?

Reviewer_5vj1OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies cost-aware active feature acquisition in multi-label data streams with delayed feedback and label-specific concept drift. The authors propose DriftBudget, a framework that estimates label-wise drift probabilities from delayed prediction residuals and uses them to weight the expected reduction in class-balanced prediction risk that each candidate feature provides for each label. The resulting cost-normalized value-of-information score guides feature acquisition under a limited budget, while delayed credit assignment updates feature-value estimates when ground-truth labels become available. Experiments on the chronological MultiEURLEX dataset report improvements compared with existing active feature acquisition baselines at comparable acquisition costs.

Critical strengths
  1. Relevant problem setting. The paper considers a practically relevant setting that combines cost-aware feature acquisition, label-specific concept drift, and delayed feedback.

  2. Simple and interpretable scoring rule. The acquisition score (Eq. 4) is easy to understand: label-wise drift posteriors weight label-specific risk reduction, normalized by cost.

  3. Promising empirical results. The reported MultiEURLEX results show improvements over existing active feature acquisition baselines at comparable acquisition costs.

Critical weaknesses
  1. Insufficient empirical evidence. Quantitative results are reported for only one real-world stream (MultiEURLEX), and most results are presented in a single table. The synthetic P0 experiments and the MIMIC-III evaluation are described in Section 6, but no results are provided. Several key endpoints registered in Table 1 are also missing, as the authors acknowledge in Section 8, including the rank correlation between predicted and realized feature value, which is the most direct test of the proposed mechanism, the equivalence analysis under stationary or global drift (H2), and the misspecification and no-support stress tests (H4).

  2. Key baseline missing. Section 2 identifies global-drift DiFA as the nearest mechanism control, yet it does not appear in Table 2. Several other baselines listed as required in Section 6 are also absent, including explainability-driven AFA, static cost-aware acquisition, and the oracle drift-boundary detector. Moreover, the costs of the ablation variants discussed in Section 7 (e.g., replacing the label-wise posterior with a global drift score) are not reported, so it is unclear whether they are compared at matched cost.

  3. Insufficient methodological and experimental details. Several essential components are not described in sufficient detail for readers to understand or reproduce the approach. For example, Section 4.1 introduces a state-space filter for estimating label-wise drift probabilities without specifying its precise formulation or update procedure. Similarly, the conditional feature model used to estimate expected risk reduction is not adequately explained. The experimental setup is also underspecified: it is unclear what the optional features are on MultiEURLEX, how feature costs and label delays are assigned, how recovery is measured when the true drift points are unknown, and whether all methods share the same predictor and update procedure.

Questions for authors
  1. The authors are encouraged to provide more details about the experimental setup and implementation during the discussion period, and to include them in the revised version. In particular, it would be helpful to clarify: how the optional features, feature costs, and label delays are constructed on MultiEURLEX? How recovery is measured when the true drift points are unknown?

  2. Additional experiments and analyses, particularly on label-wise drift estimation and feature acquisition behavior, would help strengthen the paper’s empirical evidence. For example: Results on the synthetic P0 stream, where ground-truth risk reduction is available. The rank correlation between predicted and realized feature value registered in Table 1. A direct comparison with global-drift DiFA, which Section 2 identifies as the nearest mechanism.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.