acceptodds
Under review as a conference paper at ICLR 2027

The Memory Horizon Is the Lever: Benchmarking Online Active Learning under Concept Drift and Label Delay

Abstract

Active learning (AL) is studied almost entirely as an acquisition problem, and benchmarks on stationary data rank uncertainty sampling as the strategy to beat. In streaming deployment, however, purchased labels arrive after a delay and may describe a concept that has changed by the time they arrive. We study which decision of an online learner determines accuracy under these conditions: which points to query, or how to use the delayed labels once they arrive. We build a factorial benchmark that crosses eleven acquisition strategies with ten update rules on 50 audited temporal streams at three label delays, and propose an online horizon selector that chooses how many recent labels to learn from using only labels already paid for. No tested acquisition strategy improves on random labeling by more than 0.8 points in any cell at delays up to 500 steps, and a positive control shows that the gains acquisition achieves on stationary image features vanish once the relation between features and labels drifts. The update rule moves accuracy by 9 points, and its decisive property is the memory horizon: a buffer of the 50 most recent labels scores 77.5% against 67.9% for an unbounded one, yet loses up to 9 points on real streams whose drift is slow. The selector matches a per-stream oracle at delays up to 500 steps and costs 0.5 points where nothing drifts. Code and results: https://anonymous.4open.science/r/active-learning-cd-iclr27-1342/.

Primary area:
datasets and benchmarks
Keywords:
online active learning, concept drift, label delay, memory horizon, stream benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xRuMOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper benchmarks online active learning under concept drift and label delay, comparing 11 acquisition strategies and 10 update rules across 50 temporal streams. It finds that memory horizon has a substantially larger impact on accuracy than the tested acquisition strategies. A proposed online horizon selector adapts memory length using previously purchased labels and approaches oracle performance at moderate delays.

Critical strengths
  1. Comprehensive factorial evaluation. Jointly comparing acquisition and update strategies reveals an important design consideration overlooked by conventional active learning benchmarks.

  2. Strong experimental validation. Temporal-order auditing, stationary and positive controls, memory-horizon sweeps, alternative learners, and sensitivity analyses strengthen the central empirical finding.

Critical weaknesses
  1. Limited generalizability of acquisition findings. The conclusion that acquisition offers little advantage over random sampling is based primarily on a fixed micro-batch protocol (L=100), standardized or adapted acquisition baselines, deterministic label delays, and limited budget settings. These choices improve comparability but differ from the instance-wise, variable-latency, and adaptive-budget settings targeted by several original methods. Although the stationary positive control supports the validity of the benchmark, additional sensitivity experiments are needed to establish whether the negative acquisition results generalize beyond this protocol.

  2. Statistical dependence and robustness across streams. The benchmark contains related synthetic generator variants and multiple Insects streams, potentially overstating the effective number of independent evaluation conditions. In Table 13, deduplication reduces the best update-rule improvement at delay 500 from +7.5 to +5.6 points, while none of the nine update rules passes the Holm-corrected significance test, although the bootstrap confidence interval remains positive. Family-clustered inference and separate real-versus-synthetic analyses would strengthen the statistical support and clarify the generality of the findings.

Questions for authors
  1. Micro-batch sensitivity: How do acquisition rankings change with instance-wise querying or smaller micro-batches (e.g., L=1L=1 or L=10L=10)?

  2. Baseline fidelity: How faithfully do the adapted PAL-FS and PRopagate implementations reproduce the original methods? Would variable label delays change the conclusions?

  3. Statistical robustness: Can the authors report family-clustered statistical tests to account for dependence among related streams?

  4. Horizon selection: How robust is the selector when acquisition is nonrandom, given that its error estimates use only purchased labels?

2 comments
AuthorsOpenReview ↗

Response to reviewer xRuM

Thank you for the positive assessment and the four precise questions. We answered each with new experiments. Changes are in blue in the revised PDF, and new results are in Appendix D.

Q1. Micro-batch size, delays and budgets. We added instance-wise selective sampling (L=1) and L=10 (Table 41). At L=1 every arrival is predicted and may be queried, and released labels are used before the next arrival. Random acquisition queries with probability b, and RVU runs as published. A score-based strategy queries an arrival when its score reaches the (1−b′) quantile of the last 100 scores, where b′ corrects the spending deficit. This is a balanced quantile filter, as in the PAL stream methods. SGD updates per arrival match L=100.

all 50 streamsL=1L=10L=100
best acquisition gain over random, any of 4 rules, δ=0+1.6+1.9+0.8
same, δ=500≤+0.1≤+0.1+0.2
FIFO-50 gain over naive replay, δ=0+9.5+9.6+9.4

The micro-batch protocol matters. With smaller micro-batches, margin and least-confidence sampling under A-GEM gain up to 1.9 points without delay (US 1.8). The gain is gone at δ=500, and under short-horizon rules these strategies still lose. We revised the abstract and RQ1 to say this.

Variable delays: with exponential (mean 500 and 2,000) and uniform [0, 1000] delays, labels arrive out of order. The best acquisition gain is at most +0.5 (Table 42). Budgets: with b halved or doubled, the four acquisition strategies tested there (US, margin, RVU, PAL-FS) gain at most +0.3 at δ=0 or δ=500. At δ=0, FIFO-50 gains +6.3 to +10.2 (Table 44). Adaptive budgets: RVU already spends adaptively under a budget cap, as published.

Q2. Fidelity of PAL-FS and PRopagate. PAL-FS follows Pham et al. It forgets observations that will have left the window when a candidate's label arrives, simulates labels for pending queries, and scores by the McPAL gain. PRopagate is our adaptation, as the paper states. It uses a temporally weighted soft kNN vote over labels instead of the published hard majority weighted by wall-clock time. Three new checks address fidelity.

  1. Instance-wise, as in the originals. At L=1 with the quantile filter, both stay within 0.8 points of random under every rule (PAL-FS +0.3, PRopagate −0.1 under naive replay).
  2. Tuned. We tuned the window (50 to 1,000 labels), neighbours (5 to 20), vote half-life (25 to 400) and random mixing per stream (Table 37). Neither strategy has a positive tuned mean gain over random.
  3. With PAL's native classifier. With a Parzen-window learner, PAL gains +5.3 on shuffled streams, so our implementation gains where PAL is expected to. On ordered streams PAL gains +5.0 at δ=0 but only +1.6 at δ=500 and 0.0 at δ=2000 (margin sampling: +9.7, +4.9, 0.0; Table 38).

Variable delays do not change these conclusions.

Q3. Family-clustered statistics. Table 33 treats the 31 families of the temporal set as units. The six Insects variants form one family, each generator (both variants, three realizations) forms one family, and every other real stream is its own family. Family means get equal weight, intervals resample families, and Wilcoxon tests use family means with Holm correction. Update-rule gains keep intervals above zero at δ≤500, but at δ=500 only the selector passes the corrected family-level test. No acquisition strategy has a statistically supported positive family-weighted gain, and no interval lies above zero. New Table 1 reports real (22) and synthetic (28) streams separately, plus the 12 chronologically verified real streams with their 7 families weighted equally. There, FIFO-50 gains +1.2 at δ=0 (interval −1.0 to 4.3), and online horizon selection gains +2.4 (0.3 to 5.5). We also made the audit a fixed rule. It moves HAR and Covertype out of the strict set, so the deduplicated set now has 26 streams. The corrected-test caveat you noted still holds there, and the paper states it.

Q4. Horizon selection under nonrandom acquisition. We ran the selector with all ten strategies (Table 48). Its regret to the best of its own five horizons is at most 0.3 points at δ=0, 0.9 at δ=500 and 2.4 at δ=2000. With random acquisition it is 0.2, 0.6 and 2.3. Error estimates from biased labels therefore make selection only slightly worse, although the long-delay lag remains. No strategy gains more than 0.1 points over random acquisition (RVU +0.01 at δ=0, significant but negligible; US −3.9). Expert weighting over the same buffers (Hedge, Fixed-Share) matches or beats selection, and paired learners and ADWIN trail it. In that comparison every learner buys the same examples (Table 4).

Thank you again for the constructive review.

Reviewer_xRuMOpenReview ↗

Reviewer Feedback

Thank you for the detailed rebuttal. The reported micro-batch, stochastic-delay, family-clustered, and nonrandom-acquisition experiments substantially address my original concerns. The conclusions still require qualification: acquisition retains meaningful gains with the kernel learner, and memory-selection gains are smaller and less statistically conclusive on chronologically verified real streams under delay. PRopagate also remains an adapted baseline. Overall, the factorial benchmark provides useful empirical insight, although the selector’s methodological novelty is limited. I maintain my rating 3.

Reviewer_MzszOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies online active learning when labels arrive late and the data change over time. A learner in this setting makes two choices. The first is the acquisition strategy (i.e., which points to label) and the second is the update rule (i.e., how to use the labels once they arrive). The paper asks which of these two choices matter more for accuracy. The motivation is clear and convincing. In many real applications, labels come back late. A label requested before a change may describe a pattern that no longer holds by the time it arrives. So it makes sense to ask whether choosing which points to label still matters most. The authors place their work against three areas. The first is active learning benchmarks on stable data. The second is active learning on changing data with delayed labels. The third is work on adapting to change and on continual learning, which provides change detectors, adaptive windows, and replay methods. The authors state that no earlier work tests labeling strategies and update rules together under label delay. This seems correct, and it is the clearest new point of the paper. The first main finding is that the choice of which points to label does not help. The second main finding is that the update rule matters a lot. Also a short memory is not always better.

Critical strengths
  1. The authors build a large benchmark. It tests every pairing of 11 acquisition strategies with 10 update rules. It uses 50 data streams. Of these, 22 are real and 28 come from synthetic generators.
  2. There is no existing work that test labeling strategies and update rules together under label delay.
  3. The authors show that four widely used files measure class order but not drift.
  4. Each result comes with average ranks, corrected significance tests, and bootstrap intervals. Full results for every stream are in the appendix. The authors also say clearly where their tests fail.
Critical weaknesses
  1. The 9 point figure is an average over 50 streams, 28 of them synthetic. Those streams change eight times at regular intervals, which favors short buffers by design. On the 22 real streams, the gain is about 3 points. Synthetic and real stream results should be reported separately.
  2. The selector is compared mainly with fixed buffers and with resets or gates triggered by Page Hinkley and DDM. ADWIN (Bifet and Gavaldà, 2007) is cited but never tested. This is the most important gap. ADWIN is the standard method for setting a memory window online. Its window grows while the data are stable and shrinks after a change.
  3. All main results use a linear classifier with one fixed training setup. A small neural network is checked as well. The learning rate, number of training steps, batch sizes, label budget, and delays are all fixed.
Questions for authors
  1. Please test a model for which labeling strategies clearly beat random on the shuffled tabular streams. Does that gain disappear on the ordered, delayed streams? This would directly test the authors main claim on the data the paper is mostly about.
  2. ADWIN is cited but not tested, even though it sets a memory window online. Please add an ADWIN-based update rule to the horizon comparison and to the factorial. Please test both versions: one that resets the model weights when the window is cut, and one that only shrinks the buffer.
  3. Please report the results in Table 1 separately for the 22 real streams and the 28 synthetic streams. Based on Table 4, the gain of FIFO-50 over the 2,000 label buffer appears to be about 3 points on real streams, against 14.3 on the synthetic ones.
  4. Please state the data audit as a fixed, quantitative rule with thresholds so that others can apply to new streams. This is one of the main contributions of a benchmark paper, and the rules should be reusable.
  5. The selector was run only with random labeling. The selector often picks short buffers, but it also switches to long ones on slowly changing streams, where labeling strategies might still help. Please run the labeling strategies with the selector as the update rule. This would complete the factorial for the proposed method. It would also show whether the conclusion that acquisition is not the lever still holds under the best update rule that was reported.
2 comments
AuthorsOpenReview ↗

Response to reviewer Mzsz

Thank you for the clear summary and the concrete requests. We ran all of them. Changes are in blue in the revised PDF, and new results are in Appendix D.

1. Real and synthetic streams separately (Q3). New Table 1 reports every method separately for the 22 real and 28 synthetic streams. It also reports the 12 chronologically verified real streams, with their 7 families weighted equally. The split confirms your estimate. At δ=0, FIFO-50 gains +14.2 points over the 2,000-label buffer on synthetic streams and +3.2 on real ones. On the chronological set it gains +1.2 (interval −1.0 to 4.3) and loses 1.3 at δ=500. Online horizon choice gains +2.4 there (interval 0.3 to 5.5). The abstract, introduction and conclusion now give the split instead of the pooled 9 points.

We tested both the number and the timing of changes. FIFO-50 gains +9.9 with a single change (Table 45). With eight changes it gains +14.2 when they are evenly spaced, +13.3 at random times, and +8.2 in two bursts (Table 46). Timing does matter. Regular timing favors short buffers somewhat, and bursts favor them much less. The selector follows the better rule in every case, and the appendix now says so.

2. ADWIN, both versions (Q2). We use the river implementation (δ=0.002) on the arriving-label error. ADWIN-reset wipes head and buffer when ADWIN detects an error increase. ADWIN-window keeps the head and trains only on the most recent |W| labels. Both appear in the factorial with all 11 strategies (Table 43) and in the horizon comparison (Table 4, where every learner buys the same examples):

all 50 streamsδ=0δ=500shuffled
FIFO-200067.965.968.3
ADWIN-window73.269.668.3
FIFO-5077.573.465.1
selector78.374.067.9

ADWIN detects only 3.4 changes per stream on average, so its window stays long. It gains about half as much as FIFO-50. It is, however, the only short-memory rule that costs nothing on shuffled streams. Under either ADWIN rule, no acquisition strategy has a positive mean gain over random (best −0.3).

3. Fixed training setup (W3). Table 44 changes one setting at a time: learning rate 0.1 and 1.0, 2 and 32 SGD steps, minibatch 16 and 256, and the budget halved and doubled. At δ=0, FIFO-50 gains +6.3 to +10.2 over naive replay, and the selector gains +8.1 to +10.8. The four acquisition strategies tested there (US, margin, RVU, PAL-FS) gain at most +0.3 at δ=0 or δ=500 under any of four update rules. We also added three stochastic delay distributions: exponential with mean 500 and 2,000, and uniform on [0, 1000] (Table 42). The best acquisition gain there is at most +0.5, and the selector stays within 0.7 points of the oracle. With L=1 or L=10 (Table 41), margin and least-confidence sampling under A-GEM gain up to 1.9 at δ=0 (US 1.8) but nothing at δ=500. The abstract now reports this.

Q1. A model for which acquisition beats random on shuffled tabular streams. We use a Parzen-window classifier, the model behind the probabilistic active-learning gain. Margin sampling gains +7.6 points on shuffled streams (wins/ties/losses 48/0/2). On ordered streams the gain persists without delay (+9.7) and shrinks with delay (+4.9 at δ=500, 0.0 at δ=2000). Under a 50-label buffer it becomes a loss (−7.4 at δ=500; Table 38). Drift alone does not remove the gain; delay and a short memory do. On CIFAR features the gains survive drift that moves the inputs. Drift that reassigns labels removes or reverses them, except for a two-class swap under A-GEM (+1.1 at δ=0; Table 3).

Q4. The audit as a reusable rule. Section 3.1 states three rules, and Table 30 gives the statistics. Let r be the number of label transitions divided by its expectation under a random order. R1 excludes streams with r < 0.01 that have no time-order evidence (class blocks). R2 removes a stream from the strict set if r < 0.2 and repeating the previous label is as accurate as the best learner. R3 marks a stream as chronologically verified if its source documents the order. It also accepts a stream whose embedded time field is non-decreasing within its cycles on at least 99.9% of consecutive rows. Both thresholds sit in wide gaps of r. R2 also flags HAR and Covertype, which we had kept in the strict set. The strict set now has 45 streams and the deduplicated set 26.

Q5. Labeling strategies with the selector. We ran all ten strategies with the selector (Table 48). None gains more than 0.1 points over random acquisition. The best is RVU at +0.01 at δ=0, significant but negligible (US: −3.9). The selector's regret to the best of its own horizons is at most 0.3 at δ=0, 0.9 at δ=500 and 2.4 at δ=2000. With random acquisition it is 0.2, 0.6 and 2.3. Biased labels make selection only slightly worse, and the long-delay lag remains. The conclusion holds under the selector.

We hope these additions address your concerns and are glad to clarify any point.

Thank you for the detailed response. Given these changes, I am changing my rating to 3: weak acceptance. My choice of weak acceptance is mainly due to the limited novelty and impact of this manuscript.

Reviewer_oGtUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper is a benchmark approach to analyzing different online active learning strategies under concept drift and delayed labels. It analyzes acquisition strategies with different model update rules. It also proposes a selector that chooses from five memory horizons with prequential errors on acquired class labels.

Critical strengths

I commend the Authors for factorial design that addresses an important confound in active-learning evaluation. The experimental effort is substantial, including stationary and positive controls, an MLP replication, implementation corrections, and sensitivity analyses on deduplicated streams.

Critical weaknesses

I found three major weaknesses of this work that prevent me from recommending this paper for publication:

W1: The conclusions are broader than the evidence support. We can see that acquisition already fails on the shuffled tabular controls, so those results cannot attribute its failure specifically to drift or delay. The positive image-feature control is valuable, but I see that conditional changes are implemented through label swaps and permutations, which cover a restricted form of drift. Table 2 contains statistically significant positive gains, including +0.8 points, contradicting categorical statements that no strategy improves over random. In my opinion the supported conclusion concerns small aggregate gains under the tested protocols; per-stream benefits and method-specific tuning require closer examination.

W2: The audit does not establish that all retained datasets are temporal streams. I agree that removinf class-block ordering is a useful safeguard. however in my opinion passing that test does not demonstrate chronological provenance. Table 29 still lists several datasets only in file order, including Poker, HIGGS, and Phishing. Furthermore, 28 synthetic variants and six Insects variants strongly influence the aggregate results. The deduplicated analysis helps, but corrected significance weakens under delay. A primary analysis restricted to independently sourced, chronologically verified streams is needed to substantiate the deployment claims.

W3: The selector’s added value is not sufficiently established against SOTA adaptive alternatives. Authors extend the established stable/reactive learner idea to a horizon grid, but in my opinion there is a vital lack of direct comparisons with paired learners or adaptive expert weighting under the same delayed-label protocol. It requires approximately five times the compute and includes an unbounded-memory learner, while improving over the best fixed horizon by only 0.7 and 0.4 points at delays 0 and 500. Its deficit at delay 2,000 and comparison against a best-fixed-horizon oracle further limit the “oracle matching” claim. Resource-matched comparisons would clarify whether this is a useful methodological contribution or primarily a practical benchmark baseline.

Questions for authors

I would like the authors to address both these questions and weaknesses W1-W3:

Q1: Which streams show meaningful acquisition gains, and do the conclusions persist with independently tuned acquisition methods and conditional drift beyond label swaps or permutations?

Q2: Can you please report the main factorial results on only chronologically verified real streams, weighting independent dataset families equally and providing uncertainty for the acquisition–update differences?

Q3: How does the selector compare with paired learners and adaptive expert weighting under matched label, memory, and computation budgets, particularly at long or stochastic delays?

3 comments
AuthorsOpenReview ↗

Response to reviewer oGtU

Thank you for the detailed review. Several of our claims were broader than the evidence. The revision narrows them and adds the experiments you asked for. Changes are in blue in the revised PDF, and new results are in Appendix D.

W1 / Q1. Scope of the acquisition claim

  • Shuffled controls. Our linear head fails on shuffled tabular streams, so those streams cannot show that drift or delay causes the failure. We added a Parzen-window classifier, the model of the PAL gain, for which acquisition works there. Margin sampling gains +7.6 points on shuffled streams. On ordered streams it keeps +9.7 at δ=0, falls to +4.9 at δ=500 and 0.0 at δ=2000, and becomes −7.4 under a 50-label buffer at δ=500 (Table 38).
  • Drift beyond swaps and permutations. Seven new CIFAR conditions move the inputs and keep the labels (rotations, class-mean shifts, subpopulation shift). In all of them, entropy sampling keeps +2.3 to +6.4 points at δ=0 under FIFO-50 or A-GEM. Drift that reassigns labels removes or reverses the gain in all but one setting: a two-class swap under A-GEM keeps +1.1 at δ=0 (−1.4 at δ=500; Table 3). We withdrew "the gain vanishes once p(y|x) drifts". We also corrected "prior shift leaves p(y|x) fixed"; it leaves p(x|y) fixed.
  • Wording. The abstract, Figure 1, results and conclusion now say "no strategy gains more than 0.8 points on average". The text names the small significant gains (RVU +0.1 to +0.3, split +0.8 under A-GEM).
  • Per-stream gains. At δ=0, 402 of 3,000 stream×strategy×rule cells gain more than 1 point and 2 standard errors, and 1,148 lose as much (Table 36).
  • Tuned acquisition. We tuned 44 configurations of the ten strategies per stream on the first 25% of each stream and scored the rest. The best gain is +1.6 at δ=0 (margin and least-confidence sampling mixed with random queries, under A-GEM; US +1.5) and +0.4 at δ=500. Smaller micro-batches give at most +1.9 at δ=0 (US +1.8). The FIFO capacity tuned the same way gains +11.1 and +9.0 (Table 37).

W2 / Q2. Chronology, weighting, uncertainty

  • The audit is now a fixed rule with thresholds (Sec. 3.1, Table 30). Poker, HIGGS, Phishing, Shuttle, CTG, HAR, Nomao and Covertype have no time-order evidence and leave the primary analysis. We checked Airlines and Bank for global order. Airlines covers 8.7 consecutive days in weekday order, with departure time non-decreasing within each day. Bank's month runs follow the calendar from May 2008 to November 2010, as the source documents. Removing both moves the selector's chronological-set gain from +2.4 / +1.1 to +3.3 / +1.5 (Table 32). R2 also flags HAR and Covertype, which we had wrongly kept in the strict set.
  • The primary real analysis uses 12 chronologically verified streams in 7 families, with families weighted equally and family-cluster bootstrap intervals (new Table 1, next to real and synthetic):
chronological set, Δ vs referenceδ=0δ=500
FIFO-50+1.2 [−1.0, 4.3]−1.3
online horizon selection+2.4 [0.3, 5.5]+1.1 [−0.6, 3.8]
selection minus best acquisition strategy+2.5 [0.3, 5.7]+1.1 [−0.5, 3.9]

In the full acquisition × update factorial on this set, no acquisition gain has an interval above zero (Table 34). Contrasts with fixed short horizons include zero, except that A-GEM falls below acquisition at δ=2000 (Table 35). The 9-point headline came from the generators (+14.2 synthetic, +3.2 real), and the abstract and conclusion now say so.

W3 / Q3. Selector versus adaptive alternatives (Table 4)

Our first comparison did not use the same labels. Queries and SGD shared one random generator, so learners with different numbers of heads bought equally many but different examples. Queries now have their own generator. A trace confirms identical examples for every learner, and we reran the table:

learnerδ=05002000exp(500)exp(2000)
best single fixed horizon77.673.563.874.869.4
selector78.374.062.875.169.5
Hedge, same buffers78.274.264.275.369.9
Fixed-Share78.674.163.275.470.0
paired learners75.870.760.471.865.8
FIFO-50, 5× SGD updates77.873.563.774.769.2
  • Expert weighting does as well or better and removes the long-delay deficit. The contribution is choosing the horizon online, not the argmin rule, which we now present as a benchmark baseline.
  • Dropping the unbounded buffer changes nothing (78.3 / 74.0 / 62.9). Giving FIFO-50 the selector's number of SGD updates adds at most 0.4. The selector still predicts with five heads, so its compute is about five times that of one learner.
  • "Oracle" is renamed "best fixed horizon chosen per stream in hindsight".

We hope the narrower claims and new results address your soundness concerns, and would be grateful if you reconsidered your score.

Thank you so much for your rebuttal. However, I am still of opinion that this is not work with enough novelty to justify ICLR presentation. I decided to maintain my score.

AuthorsOpenReview ↗

2nd response to reviewer oGtU

Thank you for reading our rebuttal and for replying. We take the novelty concern seriously. Could you point us to the prior work you see as overlapping? We would cite it and state the difference in the paper.

To make the new content easy to check, here are three results we could not find in earlier work. All three come from the factorial design you named as a strength.

1. The two decisions interact, and the best choice from each literature hurts when combined. Uncertainty sampling loses 1.5 points to random under naive replay. Under FIFO-50 it loses 5.5, and under a detector gate 6.0 (Table 2). With a Parzen learner, margin sampling gains 9.7 points without delay. Under a 50-label buffer at δ=500 it loses 7.4 (Table 38). Stationary AL benchmarks fix the memory policy, and stream AL papers fix the update rule. Neither design can show this interaction.

2. The kind of drift, not its size, decides whether acquisition survives. On CIFAR features, a class-mean shift that changes 28% of predictions keeps a gain of +2.8 to +5.1 points across three update rules. A two-class swap that changes only 17% cuts it to between −1.3 and +1.1 (Table 3). Under FIFO-50 or A-GEM, all eight drifts that keep the labels keep a gain of +2.3 to +6.4. Drift that relabels removes or reverses it, except the swap under A-GEM.

3. Common benchmark files measure class order, not drift. MAGIC, Skin, Spam and KDD99 are ordered in class blocks. Only 12 of our 26 real streams have verifiable chronology. We give a fixed audit rule that others can apply to new streams (Sec. 3.1, Table 30).

Your review raised W1 to W3 on soundness. If any of them is still open after the revision, we would be glad to address it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.