- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper benchmarks online active learning under concept drift and label delay, comparing 11 acquisition strategies and 10 update rules across 50 temporal streams. It finds that memory horizon has a substantially larger impact on accuracy than the tested acquisition strategies. A proposed online horizon selector adapts memory length using previously purchased labels and approaches oracle performance at moderate delays.
Critical strengths
-
Comprehensive factorial evaluation. Jointly comparing acquisition and update strategies reveals an important design consideration overlooked by conventional active learning benchmarks.
-
Strong experimental validation. Temporal-order auditing, stationary and positive controls, memory-horizon sweeps, alternative learners, and sensitivity analyses strengthen the central empirical finding.
Critical weaknesses
-
Limited generalizability of acquisition findings. The conclusion that acquisition offers little advantage over random sampling is based primarily on a fixed micro-batch protocol (L=100), standardized or adapted acquisition baselines, deterministic label delays, and limited budget settings. These choices improve comparability but differ from the instance-wise, variable-latency, and adaptive-budget settings targeted by several original methods. Although the stationary positive control supports the validity of the benchmark, additional sensitivity experiments are needed to establish whether the negative acquisition results generalize beyond this protocol.
-
Statistical dependence and robustness across streams. The benchmark contains related synthetic generator variants and multiple Insects streams, potentially overstating the effective number of independent evaluation conditions. In Table 13, deduplication reduces the best update-rule improvement at delay 500 from +7.5 to +5.6 points, while none of the nine update rules passes the Holm-corrected significance test, although the bootstrap confidence interval remains positive. Family-clustered inference and separate real-versus-synthetic analyses would strengthen the statistical support and clarify the generality of the findings.
Questions for authors
-
Micro-batch sensitivity: How do acquisition rankings change with instance-wise querying or smaller micro-batches (e.g., or )?
-
Baseline fidelity: How faithfully do the adapted PAL-FS and PRopagate implementations reproduce the original methods? Would variable label delays change the conclusions?
-
Statistical robustness: Can the authors report family-clustered statistical tests to account for dependence among related streams?
-
Horizon selection: How robust is the selector when acquisition is nonrandom, given that its error estimates use only purchased labels?
2 comments
Reviewer Feedback
Thank you for the detailed rebuttal. The reported micro-batch, stochastic-delay, family-clustered, and nonrandom-acquisition experiments substantially address my original concerns. The conclusions still require qualification: acquisition retains meaningful gains with the kernel learner, and memory-selection gains are smaller and less statistically conclusive on chronologically verified real streams under delay. PRopagate also remains an adapted baseline. Overall, the factorial benchmark provides useful empirical insight, although the selector’s methodological novelty is limited. I maintain my rating 3.
Response to reviewer xRuM
Thank you for the positive assessment and the four precise questions. We answered each with new experiments. Changes are in blue in the revised PDF, and new results are in Appendix D.
Q1. Micro-batch size, delays and budgets. We added instance-wise selective sampling (L=1) and L=10 (Table 41). At L=1 every arrival is predicted and may be queried, and released labels are used before the next arrival. Random acquisition queries with probability b, and RVU runs as published. A score-based strategy queries an arrival when its score reaches the (1−b′) quantile of the last 100 scores, where b′ corrects the spending deficit. This is a balanced quantile filter, as in the PAL stream methods. SGD updates per arrival match L=100.
The micro-batch protocol matters. With smaller micro-batches, margin and least-confidence sampling under A-GEM gain up to 1.9 points without delay (US 1.8). The gain is gone at δ=500, and under short-horizon rules these strategies still lose. We revised the abstract and RQ1 to say this.
Variable delays: with exponential (mean 500 and 2,000) and uniform [0, 1000] delays, labels arrive out of order. The best acquisition gain is at most +0.5 (Table 42). Budgets: with b halved or doubled, the four acquisition strategies tested there (US, margin, RVU, PAL-FS) gain at most +0.3 at δ=0 or δ=500. At δ=0, FIFO-50 gains +6.3 to +10.2 (Table 44). Adaptive budgets: RVU already spends adaptively under a budget cap, as published.
Q2. Fidelity of PAL-FS and PRopagate. PAL-FS follows Pham et al. It forgets observations that will have left the window when a candidate's label arrives, simulates labels for pending queries, and scores by the McPAL gain. PRopagate is our adaptation, as the paper states. It uses a temporally weighted soft kNN vote over labels instead of the published hard majority weighted by wall-clock time. Three new checks address fidelity.
Variable delays do not change these conclusions.
Q3. Family-clustered statistics. Table 33 treats the 31 families of the temporal set as units. The six Insects variants form one family, each generator (both variants, three realizations) forms one family, and every other real stream is its own family. Family means get equal weight, intervals resample families, and Wilcoxon tests use family means with Holm correction. Update-rule gains keep intervals above zero at δ≤500, but at δ=500 only the selector passes the corrected family-level test. No acquisition strategy has a statistically supported positive family-weighted gain, and no interval lies above zero. New Table 1 reports real (22) and synthetic (28) streams separately, plus the 12 chronologically verified real streams with their 7 families weighted equally. There, FIFO-50 gains +1.2 at δ=0 (interval −1.0 to 4.3), and online horizon selection gains +2.4 (0.3 to 5.5). We also made the audit a fixed rule. It moves HAR and Covertype out of the strict set, so the deduplicated set now has 26 streams. The corrected-test caveat you noted still holds there, and the paper states it.
Q4. Horizon selection under nonrandom acquisition. We ran the selector with all ten strategies (Table 48). Its regret to the best of its own five horizons is at most 0.3 points at δ=0, 0.9 at δ=500 and 2.4 at δ=2000. With random acquisition it is 0.2, 0.6 and 2.3. Error estimates from biased labels therefore make selection only slightly worse, although the long-delay lag remains. No strategy gains more than 0.1 points over random acquisition (RVU +0.01 at δ=0, significant but negligible; US −3.9). Expert weighting over the same buffers (Hedge, Fixed-Share) matches or beats selection, and paired learners and ADWIN trail it. In that comparison every learner buys the same examples (Table 4).
Thank you again for the constructive review.