- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies active feature acquisition for multi-label streams with delayed supervision and label-local drift. DriftBudget combines a posterior estimated from arrived label residuals with the expected label-specific reduction in class-balanced predictive risk, normalizes the resulting score by feature cost, and greedily acquires feasible features. Delayed labels subsequently update the drift state and a timestamped feature-value model. The reported empirical evidence consists of a chronological MultiEURLEX comparison and several aggregate posterior, global-state, delayed-credit, and delay controls; a broader synthetic and multi-dataset protocol is described but its principal mechanism endpoints are not reported.
Critical strengths
The problem is general and well-motivated: delayed feedback and heterogeneous label drift can make aggregate uncertainty an inadequate guide to purchasing measurements, particularly for rare labels.
Critical weaknesses
-
This paper is not a research paper, but a technical report, and the central method is not specified at an implementable level. Eq. (2) defines a posterior but Section 4.1 gives neither a transition model nor a residual likelihood, prior, propagation rule, or concrete calibration procedure. Sections 4.2-4.3 likewise do not specify the conditional feature distribution, predictive-risk estimator, or feature-value model update. The loss, label weights, shrinkage, beta, stopping threshold, and predictor architecture are not instantiated.
-
The evidence required to establish the claimed interaction is largely missing. Table 1 specifies heterogeneous/global drift comparisons, predicted-versus-realized value rank correlation, and misspecification/support tests; Sections 6 and 8 also name stationary equivalence and posterior-localization endpoints. None is quantitatively reported.
-
The core score in Eq. (4) can undermine Eq. (1)’s risk–cost objective even with perfect feature-value estimates. With equal costs and label weights, beta = 0.1, and budget for one feature, risk reductions of 0.20 and 0.10 become scores proportional to 0.02 and 0.10 for drift posteriors 0 and 1, respectively. The policy selects the smaller benefit at identical cost. Please derive this multiplier from a decision model and explain why Eq. (3)’s current risk estimates need additional drift weighting.
Questions for authors
see weaknesses