- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies rare-class discovery in active learning settings where some queried images cannot be assigned a usable class label, termed "failed picks." The central idea is to retain failed picks as distance anchors in greedy k-center CoreSet rather than simply removing them from future selection. This lowers the priority of nearby samples while keeping them eligible.
The main confirmatory result is a preregistered comparison on iWildCam 2022 and Snapshot Serengeti using frozen DINOv3 embeddings. Anchored CoreSet improves discovery AUC over skip-only CoreSet by 9.5% and 32.3%, respectively, while skip-only CoreSet also substantially outperforms random selection. However, only one of the three preregistered hypotheses is confirmed. The broader claim that distance-seeking methods consistently outperform density-seeking methods is not confirmed, and the predicted degradation of CoreSet under re-proposal is also not observed.
The paper additionally provides exploratory analyses across multiple selectors, encoders, pool conventions, learned labelability filtering, MegaDetector filtering, and synthetic interventions. A key empirical observation is that the treatment of unlabelable samples can substantially alter conclusions about active-learning performance. The paper also releases the "labelfirst" framework for evaluating these conventions.
Critical strengths
-
Strong experimental and statistical discipline. The use of preregistered hypotheses, paired experimental designs, exact sign-flip tests, Holm correction, multiple seeds, and explicit separation between confirmatory and exploratory analyses is a major strength. This substantially increases confidence that the central H2 result is not the result of post hoc analysis.
-
A practical issue that is often ignored in active learning. Many active-learning formulations implicitly assume that every query produces a useful label. The paper demonstrates that the treatment of failed queries can substantially alter selector rankings, including changing CoreSet from below random to substantially above random in some experimental settings. This is an important implementation detail with practical consequences.
-
Broad empirical auditing rather than comparison only against random selection. The exploratory experiments consider a reasonably broad collection of acquisition strategies, including CoreSet, LOF, kNN-based methods, BADGE, TypiClust, ProbCover, MaxHerding, k-means++, and others. The additional encoder, detector-filtering, and pool-convention experiments help reveal where the central observation does and does not hold.
-
Good transparency about negative results and limitations. The paper does not overstate the preregistered findings. Two of the three hypotheses are explicitly reported as not confirmed, and the paper documents cases where the proposed intuition becomes weaker or even reverses. This makes the empirical contribution more credible.
Critical weaknesses
-
Limited methodological novelty. The main proposed intervention is very simple: retain an inspected failed sample as a CoreSet distance anchor. The paper explicitly states that CoreSetaf is not a new selection algorithm and can coincide with standard CoreSet behavior under an appropriate treatment of inspected samples. Therefore, the main contribution is largely an empirical observation and evaluation protocol rather than a new active-learning method. For a venue such as ICLR, this makes the significance of the methodological contribution somewhat borderline.
-
The strongest confirmed claim has relatively narrow external validity. The only confirmed preregistered hypothesis is H2, evaluated on two camera-trap datasets using one preregistered DINOv3 representation and unfiltered pools. The broader distance-versus-density hypothesis is not confirmed, and the result depends on dataset geometry and representation. More importantly, after MegaDetector filtering, anchoring becomes worse than skip-only CoreSet on iWildCam while remaining better on Serengeti. This makes it difficult to formulate a general practical recommendation beyond the specific tested setting.
-
Insufficient qualitative evidence for the proposed mechanism. The paper is about image selection, yet it provides essentially no qualitative visualization of what the acquisition strategies actually select. It would be highly informative to show failed picks, their nearest neighbors in embedding space, and examples of subsequent samples selected by anchored versus skip-only CoreSet. Without this, the explanation that failed picks identify regions that should be considered covered remains mostly geometric and statistical rather than visually demonstrated.
-
Several experiments rely on simplified or simulated annotation behavior. The drone-pool failed picks are simulated because the unlabeled images were never actually presented to annotators, and the simulator assumes a fixed pool and perfect oracle. These experiments are useful for analysis but are weaker evidence for real annotation workflows. This matters because the paper is motivated primarily as a practical annotation-budget problem.
Questions for authors
-
Clarification regarding the intended novelty. The paper states that CoreSetaf is not a new selection algorithm and that standard CoreSet can make identical selections under an appropriate convention for inspected samples. Could the authors clarify more precisely what they consider the main algorithmic contribution, as opposed to the empirical and evaluation contribution? This distinction materially affects my assessment of significance.
-
Clarification regarding deployment after filtering. MegaDetector filtering improves overall discovery but reverses the advantage of anchoring over skip-only CoreSet on iWildCam. Since practical camera-trap pipelines may already use an empty-frame or animal detector, what concrete recommendation should a practitioner take from the paper regarding whether failed picks should be retained as anchors after such filtering?
-
Clarification regarding the definition of rarity. The main formulation defines rare classes as classes with at most five labelable examples in the pool. How sensitive is the H2 anchoring advantage to this threshold? Evidence that the conclusion persists across substantially different rarity thresholds would strengthen the generality of the main claim.
-
Suggestion that would affect interpretability of the claimed mechanism. Could the authors provide qualitative examples showing failed picks, their nearest neighbors, and the samples selected by anchored and skip-only CoreSet? Such examples would help determine whether the observed gain corresponds to semantically coherent failure regions or is primarily an artifact of the embedding geometry.
The paper is carefully executed and the empirical findings are credible, but I currently lean toward weak rejection because the methodological novelty is limited. The main intervention consists of retaining failed annotation queries as distance anchors within an existing CoreSet procedure, rather than introducing a substantially new acquisition method.
I find the empirical observation regarding failed-pick handling useful, and the preregistered evaluation is a clear strength. However, given that only the narrower H2 hypothesis is confirmed and the anchoring advantage is not consistent across all experimental conditions, I am not yet convinced that the contribution is sufficiently novel or broadly significant for acceptance.
A strong rebuttal that better establishes the novelty relative to existing CoreSet implementations and clarifies the broader significance and applicability of the finding could change my assessment.
2 comments
Thank you for the detailed response and for addressing the concerns with additional analyses and experiments. The new results appear helpful and directly relevant to several of my questions. I will read the rebuttal and the added material more carefully and reconsider my score accordingly.
Novelty, use after detector filters, rarity thresholds
Added for this review: a rarity-threshold recount (Appendix V.6, p. 58), the example figures (Figure 3, p. 8; Figures V.1 and V.2, p. 51 and 52) and the rule for pools that a detector has filtered (Table V.5, p. 61).
Short answers
Details
Q1 and Weakness 1. Intended novelty. The paper claims no new selection algorithm, and it is submitted to the datasets and benchmarks area. On pools with unlabelable items, active-learning results depend on how a tool treats a failed pick, and tools treat it differently (general response; Table T.1, p. 47). In exploratory drone-pool runs the choice moves CoreSet from 27% below random to 46% above. Four tools re-propose failed picks (scikit-activeml, libact, LfOSA, Label Studio), four remove them or set them aside (modAL, small-text, CCAL, MQNet), and none keeps a failed pick as a plain farthest-first anchor (Table O.2, p. 40). So the same selector runs under a different pool convention depending on the library. The significance: results on such pools cannot be compared unless a paper states its pool convention. The paper supplies the audit, and the labelfirst library reports results under each convention.
Q2. Use after a detector filter. Keep failed picks as anchors after a filter too: across 58 paired comparisons, always anchoring loses least (general response, "When to anchor"; Table V.5, p. 61). Anchoring hurt only on iWildCam after a MegaDetector filter, by up to 13.5 discovery AUC (Table V.4, p. 59). On filtered versions of two further pools it helped in 14 of 18 comparisons and never hurt (Appendix V.7, p. 58).
Q3. Rarity threshold. The gain persists at thresholds 2, 5 and 10 in an exploratory recount (Appendix V.6, p. 58):
Q4 and Weakness 3. Qualitative examples. The failure regions are semantically coherent: they are empty frames. Of an unlabelable frame's ten nearest neighbors, 0.960 (iWildCam) and 0.973 (Serengeti) are unlabelable (Table V.1, p. 50). In Figure 3 (p. 8), row (a) is an empty frame that the loop opened, and its five nearest unopened candidates are all empty. Row (c) is a tiger, and four of its five nearest candidates are tigers. Row (b) is a bad case: a visible bird that the dataset labels as empty, with five animal frames as its neighbors. A fixed rule chose the rows. Appendix V.4 (p. 55) replays single rounds with the anchor on and off.
Weakness 2. External validity. The confirmed claim is still H2 on two datasets with one preregistered encoder, and exploratory reruns agree: two more encoders (Table V.3, p. 59; Appendix Q, p. 43) and two pools built after the reviews, Idaho +10.4 [+7.6, +13.1] and WSU Lynx +11.9 [+10.6, +13.4] discovery AUC (Appendix V.7). The gain does not extend to every selector: a benchmark of 14 selectors failed both of its registered tests, so the Q2 rule is stated for CoreSet (Appendix V.8, p. 61).
Weakness 4. Simulated failures and a perfect oracle. Only the drone pool has simulated failures, so it is exploratory. On camera traps an empty frame is a recorded annotation result. With an imperfect oracle that reports 5, 10 or 20% of labelable picks as unlabelable, the gain stays at +8.2% or more on iWildCam and +28.2% or more on Serengeti (Table V.3). Orinoquia has a label for each frame, and there anchoring gains +2.94 [+1.44, +4.31] discovery AUC. With sequence labels on the same frames the gain falls to +0.44 [-0.44, +1.31], so sequence labels hide the gain; they do not create it (Appendix E, p. 26).
You asked for a clearer case on novelty against existing CoreSet implementations, on significance and on applicability. Q1 answers the first two and Q2 the third.