acceptodds
Under review as a conference paper at ICLR 2027

Which Samples to Label First? Testing Failed Picks as Distance Anchors for Rare-Class Discovery

Abstract

Annotators searching image archives for rare species, from drone surveys of forest canopy to camera traps, often open frames that cannot be given any class label, such as empty ones, which costs budget; we call this a failed pick. A failed pick gives no class label, but remembering where it lies can still help find rare classes. Anchored k-center CoreSet keeps each failed pick as a distance anchor, lowering nearby images' priority without removing them. In preregistered tests on iWildCam and Snapshot Serengeti, anchoring raises discovery AUC, distinct rare classes found summed over ten rounds, by +9.5% and +32.3% over skipping alone. CoreSet that skips failed picks also beats random selection that skips them; all 16 seeds favor each of the four comparisons (exact sign-flip p = 2^-16 on each). This confirms one of three preregistered hypotheses. The two broader ones are not confirmed: the pooled distance-versus-density contrast misses significance and reverses on one of four datasets, and under re-proposal CoreSet stays ahead of random on both camera-trap datasets, opposite to the prediction. In exploratory drone-pool experiments with simulated failures, the pool convention, how a tool treats unlabelable items, moves CoreSet from 27% below random to 46% above, and an audit of 19 tools finds no common convention. On that pool, learned labelability filtering avoids more failed picks yet has lower discovery AUC than anchoring on two encoders; the third shows no significant difference in this endpoint. In an exploratory MegaDetector comparison, detector filtering raises mean discovery AUC for all three selectors but makes some rare classes unreachable; it reverses the anchoring advantage on iWildCam but not on Serengeti. We contribute evidence, not a new CoreSet algorithm, and release labelfirst, a library that audits rare-class discovery under each pool convention.

Primary area:
datasets and benchmarks
Keywords:
active learning, rare-category detection, failed picks, distance anchors, coreset, preregistration, statistical hypothesis testing, long-tailed recognition, foundation model embeddings, camera traps, biodiversity monitoring

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HVPPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies rare-class discovery in active learning settings where some queried images cannot be assigned a usable class label, termed "failed picks." The central idea is to retain failed picks as distance anchors in greedy k-center CoreSet rather than simply removing them from future selection. This lowers the priority of nearby samples while keeping them eligible.

The main confirmatory result is a preregistered comparison on iWildCam 2022 and Snapshot Serengeti using frozen DINOv3 embeddings. Anchored CoreSet improves discovery AUC over skip-only CoreSet by 9.5% and 32.3%, respectively, while skip-only CoreSet also substantially outperforms random selection. However, only one of the three preregistered hypotheses is confirmed. The broader claim that distance-seeking methods consistently outperform density-seeking methods is not confirmed, and the predicted degradation of CoreSet under re-proposal is also not observed.

The paper additionally provides exploratory analyses across multiple selectors, encoders, pool conventions, learned labelability filtering, MegaDetector filtering, and synthetic interventions. A key empirical observation is that the treatment of unlabelable samples can substantially alter conclusions about active-learning performance. The paper also releases the "labelfirst" framework for evaluating these conventions.

Critical strengths
  1. Strong experimental and statistical discipline. The use of preregistered hypotheses, paired experimental designs, exact sign-flip tests, Holm correction, multiple seeds, and explicit separation between confirmatory and exploratory analyses is a major strength. This substantially increases confidence that the central H2 result is not the result of post hoc analysis.

  2. A practical issue that is often ignored in active learning. Many active-learning formulations implicitly assume that every query produces a useful label. The paper demonstrates that the treatment of failed queries can substantially alter selector rankings, including changing CoreSet from below random to substantially above random in some experimental settings. This is an important implementation detail with practical consequences.

  3. Broad empirical auditing rather than comparison only against random selection. The exploratory experiments consider a reasonably broad collection of acquisition strategies, including CoreSet, LOF, kNN-based methods, BADGE, TypiClust, ProbCover, MaxHerding, k-means++, and others. The additional encoder, detector-filtering, and pool-convention experiments help reveal where the central observation does and does not hold.

  4. Good transparency about negative results and limitations. The paper does not overstate the preregistered findings. Two of the three hypotheses are explicitly reported as not confirmed, and the paper documents cases where the proposed intuition becomes weaker or even reverses. This makes the empirical contribution more credible.

Critical weaknesses
  1. Limited methodological novelty. The main proposed intervention is very simple: retain an inspected failed sample as a CoreSet distance anchor. The paper explicitly states that CoreSetaf is not a new selection algorithm and can coincide with standard CoreSet behavior under an appropriate treatment of inspected samples. Therefore, the main contribution is largely an empirical observation and evaluation protocol rather than a new active-learning method. For a venue such as ICLR, this makes the significance of the methodological contribution somewhat borderline.

  2. The strongest confirmed claim has relatively narrow external validity. The only confirmed preregistered hypothesis is H2, evaluated on two camera-trap datasets using one preregistered DINOv3 representation and unfiltered pools. The broader distance-versus-density hypothesis is not confirmed, and the result depends on dataset geometry and representation. More importantly, after MegaDetector filtering, anchoring becomes worse than skip-only CoreSet on iWildCam while remaining better on Serengeti. This makes it difficult to formulate a general practical recommendation beyond the specific tested setting.

  3. Insufficient qualitative evidence for the proposed mechanism. The paper is about image selection, yet it provides essentially no qualitative visualization of what the acquisition strategies actually select. It would be highly informative to show failed picks, their nearest neighbors in embedding space, and examples of subsequent samples selected by anchored versus skip-only CoreSet. Without this, the explanation that failed picks identify regions that should be considered covered remains mostly geometric and statistical rather than visually demonstrated.

  4. Several experiments rely on simplified or simulated annotation behavior. The drone-pool failed picks are simulated because the unlabeled images were never actually presented to annotators, and the simulator assumes a fixed pool and perfect oracle. These experiments are useful for analysis but are weaker evidence for real annotation workflows. This matters because the paper is motivated primarily as a practical annotation-budget problem.

Questions for authors
  1. Clarification regarding the intended novelty. The paper states that CoreSetaf is not a new selection algorithm and that standard CoreSet can make identical selections under an appropriate convention for inspected samples. Could the authors clarify more precisely what they consider the main algorithmic contribution, as opposed to the empirical and evaluation contribution? This distinction materially affects my assessment of significance.

  2. Clarification regarding deployment after filtering. MegaDetector filtering improves overall discovery but reverses the advantage of anchoring over skip-only CoreSet on iWildCam. Since practical camera-trap pipelines may already use an empty-frame or animal detector, what concrete recommendation should a practitioner take from the paper regarding whether failed picks should be retained as anchors after such filtering?

  3. Clarification regarding the definition of rarity. The main formulation defines rare classes as classes with at most five labelable examples in the pool. How sensitive is the H2 anchoring advantage to this threshold? Evidence that the conclusion persists across substantially different rarity thresholds would strengthen the generality of the main claim.

  4. Suggestion that would affect interpretability of the claimed mechanism. Could the authors provide qualitative examples showing failed picks, their nearest neighbors, and the samples selected by anchored and skip-only CoreSet? Such examples would help determine whether the observed gain corresponds to semantically coherent failure regions or is primarily an artifact of the embedding geometry.

The paper is carefully executed and the empirical findings are credible, but I currently lean toward weak rejection because the methodological novelty is limited. The main intervention consists of retaining failed annotation queries as distance anchors within an existing CoreSet procedure, rather than introducing a substantially new acquisition method.

I find the empirical observation regarding failed-pick handling useful, and the preregistered evaluation is a clear strength. However, given that only the narrower H2 hypothesis is confirmed and the anchoring advantage is not consistent across all experimental conditions, I am not yet convinced that the contribution is sufficiently novel or broadly significant for acceptance.

A strong rebuttal that better establishes the novelty relative to existing CoreSet implementations and clarifies the broader significance and applicability of the finding could change my assessment.

2 comments
AuthorsOpenReview ↗

Novelty, use after detector filters, rarity thresholds

Added for this review: a rarity-threshold recount (Appendix V.6, p. 58), the example figures (Figure 3, p. 8; Figures V.1 and V.2, p. 51 and 52) and the rule for pools that a detector has filtered (Table V.5, p. 61).

Short answers

  • Q1: None, and the paper claims none. Its primary area is datasets and benchmarks, and the contribution is an evaluation finding: how a tool treats a failed pick moves the same selector on the same pool by a third on iWildCam, and none of the 19 audited tools keeps a failed pick as a plain farthest-first anchor.
  • Q2: Anchor by default with CoreSet, also after a detector filter. Anchoring hurt only on iWildCam after a filter.
  • Q3: The gain stays at rarity thresholds 2 and 10.
  • Q4: The figures show images that the loop opened with their nearest candidates, bad cases included, and Appendix V.4 compares the anchored and skip arms round by round.
  • W2 and W4: Two more encoders, two new pools and an imperfect oracle give the same direction. These results are exploratory.

Details

Q1 and Weakness 1. Intended novelty. The paper claims no new selection algorithm, and it is submitted to the datasets and benchmarks area. On pools with unlabelable items, active-learning results depend on how a tool treats a failed pick, and tools treat it differently (general response; Table T.1, p. 47). In exploratory drone-pool runs the choice moves CoreSet from 27% below random to 46% above. Four tools re-propose failed picks (scikit-activeml, libact, LfOSA, Label Studio), four remove them or set them aside (modAL, small-text, CCAL, MQNet), and none keeps a failed pick as a plain farthest-first anchor (Table O.2, p. 40). So the same selector runs under a different pool convention depending on the library. The significance: results on such pools cannot be compared unless a paper states its pool convention. The paper supplies the audit, and the labelfirst library reports results under each convention.

Q2. Use after a detector filter. Keep failed picks as anchors after a filter too: across 58 paired comparisons, always anchoring loses least (general response, "When to anchor"; Table V.5, p. 61). Anchoring hurt only on iWildCam after a MegaDetector filter, by up to 13.5 discovery AUC (Table V.4, p. 59). On filtered versions of two further pools it helped in 14 of 18 comparisons and never hurt (Appendix V.7, p. 58).

Q3. Rarity threshold. The gain persists at thresholds 2, 5 and 10 in an exploratory recount (Appendix V.6, p. 58):

Threshold2510
iWildCam+12.2%+9.5%+9.1%
Serengeti+31.3%+32.3%+20.3%

Q4 and Weakness 3. Qualitative examples. The failure regions are semantically coherent: they are empty frames. Of an unlabelable frame's ten nearest neighbors, 0.960 (iWildCam) and 0.973 (Serengeti) are unlabelable (Table V.1, p. 50). In Figure 3 (p. 8), row (a) is an empty frame that the loop opened, and its five nearest unopened candidates are all empty. Row (c) is a tiger, and four of its five nearest candidates are tigers. Row (b) is a bad case: a visible bird that the dataset labels as empty, with five animal frames as its neighbors. A fixed rule chose the rows. Appendix V.4 (p. 55) replays single rounds with the anchor on and off.

Weakness 2. External validity. The confirmed claim is still H2 on two datasets with one preregistered encoder, and exploratory reruns agree: two more encoders (Table V.3, p. 59; Appendix Q, p. 43) and two pools built after the reviews, Idaho +10.4 [+7.6, +13.1] and WSU Lynx +11.9 [+10.6, +13.4] discovery AUC (Appendix V.7). The gain does not extend to every selector: a benchmark of 14 selectors failed both of its registered tests, so the Q2 rule is stated for CoreSet (Appendix V.8, p. 61).

Weakness 4. Simulated failures and a perfect oracle. Only the drone pool has simulated failures, so it is exploratory. On camera traps an empty frame is a recorded annotation result. With an imperfect oracle that reports 5, 10 or 20% of labelable picks as unlabelable, the gain stays at +8.2% or more on iWildCam and +28.2% or more on Serengeti (Table V.3). Orinoquia has a label for each frame, and there anchoring gains +2.94 [+1.44, +4.31] discovery AUC. With sequence labels on the same frames the gain falls to +0.44 [-0.44, +1.31], so sequence labels hide the gain; they do not create it (Appendix E, p. 26).

You asked for a clearer case on novelty against existing CoreSet implementations, on significance and on applicability. Q1 answers the first two and Q2 the third.

Thank you for the detailed response and for addressing the concerns with additional analyses and experiments. The new results appear helpful and directly relevant to several of my questions. I will read the rebuttal and the added material more carefully and reconsider my score accordingly.

Reviewer_9Av4OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

When experts are selecting or sorting out images from a large dataset, they waste a lot of time and budget on getting rare-class and try to discover them as early as possible. The authors proposed that different treatments to the failed picks might help improve the overall performance of the rare-class discovery. To do this, the authors used frozen image embeddings and compared two CoreSet procedures. Both of them ignore checked images and perform better than the random picking, but one of them actually uses the failed picks as the anchors for the next round selections and outperforms the other approach. This study provides an improvement on a practical problem, as it saves the budget on checking and labeling those empty images or unlabelable ones. However, the actual improvement was not quantified or presented well and diminishes the potential impact and significance of this study. In addition, lacking sample pictures of good and bad cases makes it more abstract to understand the improvement. Furthermore, I have a fundamental concern about why and how this using failed picks as anchors improves the efficiency relating to the image embedding mechanisms.

Critical strengths

This study points out a practical issue that can be overlooked when evaluating image picking, which is there is still a room for the procedure to be improved to save evaluating budgets by selecting less unusable images and picking more images more likely to be useful. The comparison between skipping failed picks and keep them as anchors is simple and understandable. The authors evaluated the different pool conventions, selectors, embedding and filtering procedures with quite considerable effort. The paired experiments provide consistent results for the anchoring advantage on the two unfiltered camera-trap pools under the registered evaluation setting. In addition, the authors distinguish preregistered and exploratory analyses, report unsuccessful hypotheses, and acknowledge that anchoring does not always help.

Critical weaknesses
  1. My biggest concern is that the paper does not sufficiently connect the embedding geometry to the actual images being selected. The authors specify DINOv3, repeat the main comparison with CLIP, and report failure-neighborhood and rare-class-neighbor statistics. The fundamental hypothesis not been demonstrated in this paper is whether the DINOv3 could help separate difference rare-classes efficiently. Although they used a second encoder and showed a similar aggregate gain, it does not explain whether the relevant distances mainly reflect animal presence, species identity, background, or some combination of these. Therefore, if the embedding itself cannot help separating different rare-class images, it is not practical to talk about using anchoring to select larger distance candidates.
  2. For imaging picking study, strangely, there is no image or illustrations of good or bad samples in the manuscript or appendix. So it does not give the readers opportunity to have a visual idea what is happening. Without such illustrations, authors could not show how this pipeline work better and even show some extreme cases it might fail. For instance, a small animal at the edge of an image could become difficult to see after resizing and center cropping. Meanwhile, background or illumination could still strongly influence the embedding. So, I suggest the authors show representative original images, the actual inputs used by the encoder, and nearby images for both successful and failed anchors. So that authors can show that more useful around the identified anchors and fewer useful candidates around the failed picks.
  3. Improvements are mainly summarized over all rare classes, so it not clear whether there is some bias like species are found earlier and which are delayed or consistently missed. Finding about two additional classes among 15 in Serengeti could be useful, but the interpretation depends on which classes benefit and how consistently they are discovered. It would be helpful to have a per-class analysis of discovery probability and first-discovery budget across seeds, including classes that are never found. So that authors could relate these outcomes to class frequency and visible image characteristics. For successful anchors, it would also be useful to distinguish nearby images of the same class, other already discovered classes, still undiscovered classes, and unlabelable images, using the uninspected candidates at the relevant round. This would help explain which discoveries the selection rule accelerates and which it postpones.
  4. Quantifying how much improvements can be achieved by their solutions is necessary. Showing quantitative improvements respect to the time that saved for experts to evaluate and computational sources saved, and so on, would be very helpful. At the final inspection budgets, keeping failed anchors adds about 3.1 rare classes on iWildCam and 2.3 on Serengeti. These are statistically supported improvements within the evaluated pools, but readers can hardly relate it to actually save budgets, unless quantified. Besides, an empty frame may be quickly identified and labeled, but identifying an unusual animal may require more time. Thus, fewer failed picks under a fixed inspection count do not directly quantify time saved. Also, the encoding and selection themselves should also be reported in time and computational cost. Overall, it helps a lot to show exactly how much effect the presented pipeline can save compared among different ways and also versus manual selection.
Questions for authors
  1. Could the authors demonstrate that the existing embeddings are effectively separating rare-classes as well as clustering same class images and unlabelable images?
  2. Could the authors provide a compact set of actual camera-trap examples showing the original image, the preprocessed encoder input, the inspection outcome, and its nearby images in the embedding space? I am particularly interested in successful rare-class anchors, empty or otherwise unlabelable anchors, and cases in which these neighborhoods overlap. Please explain how the examples are selected so that they include both favorable and unfavorable cases.
  3. To study the bias in the embeddings, could the authors report per-class discovery probability and first-discovery budget across seeds, while explicitly accounting for classes that are never found?
  4. How much expert time would the reported improvement save under a realistic annotation workflow? A limited timing exercise or an explicit range of inspection-time assumptions would already be helpful. The authors can compare discoveries at a fixed time budget, or the time needed to reach the same discovery target, while including the relevant computational costs.
7 comments
AuthorsOpenReview ↗

Separability, examples, per-class discovery, annotation time

Added for this review: a separability table (Table V.1, p. 50), example figures (Figure 3, p. 8; Figure V.1, p. 51), a per-class table (Table V.2, p. 56) and an annotation-time section (Appendix V.5, p. 55).

Short answers

  • Q1: The embedding separates animal frames from empty frames well (ROC AUC 0.993 and 0.971), and species only partly. Anchoring needs only the first.
  • Q2: The figures show images that the loop opened, with the encoder input and the nearest candidates. A fixed rule chose them, so bad cases appear too.
  • Q3: A few classes carry the gain, and some classes lose. Each arm never finds 15 iWildCam rare classes, and both arms miss 13 of those.
  • Q4: In an informal timing check, the anchored arm finds 1.86 (iWildCam) and 2.13 (Serengeti) more rare classes in the same annotation time. Expert times are still missing.

Details

Q1. Does the embedding separate the classes? Labelable from unlabelable, almost perfectly: a linear classifier on the frozen DINOv3 embeddings reaches ROC AUC 0.993 on iWildCam and 0.971 on Serengeti. Of an unlabelable frame's ten nearest neighbors, 0.960 and 0.973 are unlabelable, against pool shares of 0.463 and 0.764. Species, partly: the nearest labelable neighbor of a rare animal has the same species 0.385 and 0.407 of the time (Table V.1, p. 50).

Q2. Image examples. Figure 3 (p. 8) and Figure V.1 (p. 51, Serengeti) show opened images, the encoder input and the nearest unopened candidates. In Figure 3, row (a) is an empty frame whose five nearest candidates are all empty. Row (c) is a tiger, and four of its five nearest candidates are tigers. Row (b) is a bad case: a visible bird that the dataset labels as empty, with five animal frames as its neighbors. Cropping. The encoder sees a center crop, so an animal at the edge of the frame can be missing from the input (Appendix V.1, p. 49). Animal size and gain are only weakly related (Appendix E, p. 26). Illumination. Empty frames sit together within day frames and within night frames (Appendix V.2, p. 50). Background. The embedding does encode the camera site, but site alone does not explain why empty frames sit together: without same-site neighbors, 0.909 and 0.961 of an unlabelable frame's neighbors are still unlabelable (Appendix V.2).

Q3. Discovery per class. Table V.2 (p. 56) gives, for each rare class, how often each arm finds it and the median budget at first discovery. On Serengeti four classes carry the gain: aardvark, bat-eared fox (found in 0.94 of seeds with anchoring against 0.19 without), bushbuck (0.94 against 0.12) and hare. For the worst iWildCam classes the chance of discovery falls by 0.31 to 0.38. Near a failed pick, 0.84 (iWildCam) and 0.94 (Serengeti) of the ten nearest unopened candidates are unlabelable, and none belongs to a class not yet found (Appendix V.3, p. 53). Near a successful rare-class pick, 0.06 and 0.10 have the same class, and 0.73 and 0.70 belong to a class already found.

Q4. Expert time and compute. The table gives both forms you offered, from an informal timing check, not a user study. One non-expert author labeled 60 iWildCam frames: an empty frame took 4.4 s and an animal frame 7.3 s, a ratio of 0.60 (95% CI 0.45 to 0.79). Serengeti uses the iWildCam times. The target is the median number of rare classes the skip arm finds in ten rounds.

iWildCamSerengeti
More rare classes in the same time (Appendix V.5)+1.86+2.13
Seeds that reach the target, anchored and skip, of 1615 and 1116 and 8
Time saved where both reach it11 of about 208 min (p = 0.021)25 of about 67 min (p = 0.004)
Same-time gain is significant above a ratio of0.410.02

Row 3 and the invited session are in Appendix V.5 (p. 57). With assumed times instead (p. 55), the skip arm was 28 minutes faster on iWildCam where both arms reach the target, so the measured times reverse that result.

Limits: an invited annotator with no camera-trap experience measured a ratio of 0.55 (0.34 to 0.87). The iWildCam gain is significant only above 0.41, and that interval dips below it. Times from experts are still missing.

Compute: selecting frames for 16 seeds of one arm took 8 to 36 s. On one H100, MegaDetector took 27.1 and 32.4 minutes per pool and the DINOv3 embedding 57 and 88 s. A comparison against manual selection is still missing.

AuthorsOpenReview ↗

Timing page from Appendix V.5

The frame-timing page behind Appendix V.5 is at https://anon-frame-timing.github.io/frame-timing-check/. It shows the same 60 iWildCam frames in the same order and, at the end, reports seconds per empty frame over seconds per animal frame with a 95% CI, which can be read against the 0.41 at which the iWildCam same-time gain is significant. Nothing is sent anywhere; the results stay in the browser and can be downloaded as a CSV.

AuthorsOpenReview ↗

Second invited timing session

A second invited annotator completed the timing page (https://anon-frame-timing.github.io/frame-timing-check/). At their times the anchored arm finds fewer rare classes than the skip arm on iWildCam in the same annotation time, and still more on Serengeti.

SessionEmpty frameAnimal frameRatio (95% CI)iWildCamSerengeti
Author 14.4 s7.3 s0.60 (0.45 to 0.79)+1.86+2.13
Invited 14.3 s7.7 s0.55 (0.34 to 0.87)+1.63+2.09
Invited 21.5 s10.8 s0.14 (0.09 to 0.23)-0.88+1.02

The last two columns give how many more rare classes the anchored arm finds in the same time.

The reason is simple. The anchored arm opens fewer empty frames and more animal frames, so its picks cost more time. If an empty frame takes 1.5 s, the skip arm loses almost no time on its failed picks, and on iWildCam that outweighs the anchored arm's gain per pick. (Two empty frames with a faulty recorded time are left out; with them the ratio is 0.10 (0.04 to 0.20)).

So per minute, anchoring leads on Serengeti for all three annotators, and on iWildCam for two of three. The results per pick (Q1 to Q3) do not depend on time.

Reviewer_VcX1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates active sample-selection strategies with a focus on early discovery of rare classes under a limited annotation budget at the (potential) detriment of classifier accuracy. The authors focus on the concept of failed picks, which is when a selected stimulus is not assigned a class label (e.g. due to no animal being present in the image). They argue that skipping the stimulus and essentially "forgetting about it" misses one (potentially) powerful aspect: stimuli in the vicinity of this skipped stimulus are (potentially) likely to also not contain any relevant classes, and thus should be considered less likely for selection. They investigate a selection strategy called anchoring that affects nearby stimuli when a stimulus was skipped due to being a failed pick. They show that this improves the number of cumulatively found rare classes over just skipping or re-proposing (two other strategies investigated). However, the advantage is not universal and depends on the dataset and selection setup, as shown in the paper as well.

The authors make four contributions: (1) Their proposed anchoring strategy beats skipping-only and random strategies. (2) They show that pool conventions need to be considered/reported when evaluating active-learning algorithms, as it can substantially change the outcomes. (3) They show that avoiding failed picks does not immediately yield more rare class findings, as using either a pretrained animal detector or learning which images are labelable both reduced the number of failed picks while either making some rare classes unreachable or also yielding fewer rare classes than using their anchoring technique. (4) The python tool labelfirst, which allows researchers to compare selection strategies, test different pool conventions, and measure rare-class discovery all in a robust, reproducible way.

With respect to validation, the authors formulate three preregistered hypotheses and test them on held-out datasets. They evaluate selection strategies on established image datasets, simulating multiple rounds of sample selection and annotation. They investigate multiple strategies, including baseline strategies, to fairly evaluate their proposed technique. They investigate different image encoders, filtering methods, pool conventions, and synthetic scnearios to ensure that anchoring works and also understand when it doesn't.

Critical strengths

The paper addresses an important and potentially overlooked issue in active learning: how unsuccessful annotation attempts are handled. The authors demonstrate that pool conventions can substantially influence experimental outcomes and that failed picks can provide useful information for subsequent sample selection. I consider this particularly valuable because it highlights an implementation detail that may affect the validity and comparability of active-learning evaluations more broadly.

The authors employ preregistered hypotheses, appropriate baseline comparisons, multiple random seeds, and statistical significance testing. Importantly, they also report findings that contradict their original hypotheses and investigate conditions under which anchoring becomes less effective. This transparency strengthens my confidence in the main findings, even though their generalizability remains limited. The accompanying labelfirst library further supports reproducibility and practical adoption.

Critical weaknesses

The central hypothesis (H2) is confirmed on only two camera-trap datasets using one primary image encoder. Furthermore, the exploratory experiments demonstrate that the advantage of anchoring is not consistent across all conditions, particularly when pretrained filtering is introduced. This raises concerns about how broadly the findings can be applied and whether anchoring provides a reliable advantage in more diverse real-world settings.

The proposed anchoring strategy is based on an existing CoreSet selection mechanism rather than introducing a fundamentally new algorithm. While the authors demonstrate empirical benefits, the experiments do not conclusively establish the conditions under which anchoring is advantageous or potentially harmful. Consequently, the paper provides useful methodological insights but offers a relatively limited advance in our general understanding of rare-class discovery.

Questions for authors

On generalizability of the anchoring advantage (clarification): The confirmed advantage of anchoring is demonstrated on two unfiltered camera-trap datasets, while the exploratory MegaDetector experiments show that this advantage can reverse under different conditions. Could the authors clarify which characteristics of a dataset or its embedding space they consider most important for determining whether anchoring is beneficial? In particular, can the existing experiments provide more concrete guidance on when practitioners should prefer anchoring over simply skipping failed picks?

On distinctiveness of contribution (clarification): The authors explicitly state that anchoring does not constitute a new selection algorithm and that their existing CoreSet implementation already exhibits equivalent behavior under certain pool conventions. Could the authors clarify what they consider the most consequential new insight beyond demonstrating the importance of handling failed picks correctly? In particular, how does their empirical evidence advance our understanding of rare-class discovery beyond existing coverage-based selection approaches?

1 comment
AuthorsOpenReview ↗

When anchoring helps, novelty, more encoders and pools

Added for this review: 58 paired comparisons (Table V.5, p. 61), two more encoders, DINOv2 and CLIP (Table V.3, p. 59; Appendix Q, p. 43) and two pools built after the reviews (Appendix V.7, p. 58).

Short answers

  • Q1: What matters most is how many rare items sit next to failed picks. Without a detector filter almost none do, and anchoring helped in all 13 such comparisons. After a filter the result is mixed, and anchoring by default is still the safer choice.
  • Q2: The main insight is that the pool convention, not the selector, decides much of the result: CoreSet moves by a third on iWildCam depending on how the tool treats a failed pick, and none of the 19 audited tools keeps a failed pick as a plain farthest-first anchor.
  • W1: Reruns on two more encoders and two new pools give the same direction. They are exploratory.

Details

Q1. When does anchoring help? It helps when few rare items sit next to failed picks. Without a filter almost none do, so an anchor mostly pushes back candidates that would fail too (Appendix V.3, p. 53), and anchoring helped in all 13 paired comparisons (Table V.5, p. 61). After a MegaDetector filter on iWildCam, the share of rare items among a failed pick's ten nearest candidates rises from 0.0021 to 0.0354. All six losses are there, up to 13.5 discovery AUC (Table V.4, p. 59). This describes the results and is not a tested rule. A running loop cannot compute that share, and the warning sign it can compute does worse than always anchoring (general response, "When to anchor"), so the practical guidance is to anchor by default.

Q2. Distinctiveness. Beyond coverage-based selection, the evidence shows that a failed pick carries information: unlabelable frames sit close together, so one remembered failure steers later picks away from many others. Failed picks fall from 773 to 238 on iWildCam and from 724 to 421 on Serengeti (Appendix V.3). A detector filter also removes empty frames, but in the paper's MegaDetector comparison it puts some rare classes out of reach. Anchoring needs no detector and removes nothing from the pool. The convention numbers are in the general response (Table T.1, p. 47; Table O.2, p. 40).

W1. Two datasets, one encoder. Other encoders and pools give the same direction. The anchored arm leads the skip arm by +8.2% [+7.2, +9.2] and +14.7% [+11.1, +18.3] with DINOv2 ViT-L/14 (Table V.3, p. 59), and by +16.1% and +57.7% with CLIP ViT-L/14 (Appendix Q, p. 43). Two unfiltered pools built after the reviews, each with its prediction committed before the runs, give Idaho +10.4 [+7.6, +13.1] and WSU Lynx +11.9 [+10.6, +13.4] discovery AUC (Appendix V.7). The confirmed claim still covers two datasets and one preregistered encoder. The gain is specific to CoreSet: a benchmark of 14 selectors, registered after the confirmatory runs, failed both of its tests (Appendix V.8, p. 61).

Comments on the paper

1 comment
AuthorsOpenReview ↗

How tools treat failed picks moves results by a third to two thirds

In short: how a tool handles an image that gets no label moves the same selector on the same pool by a third on iWildCam and two thirds on Serengeti. Each reviewer reply gives short answers first, then details with page numbers in the revised PDF.

Finding. An annotator sometimes opens an image that cannot get a label, such as an empty frame. The paper calls this a failed pick. The 19 audited tools treat a failed pick in different ways (Table O.2, p. 40), and the same selector on the same pool then gives different results:

What the tool does with a failed pickiWildCamSerengeti
Re-proposes it322.642.5
Skips it392.253.6
Keeps it as an anchor429.471.2

The values are CoreSet's discovery AUC: distinct rare classes found, summed over ten rounds (Table T.1, p. 47; exploratory, and the first step also changes initialization and stopping).

Why anchoring works. Unlabelable frames sit close together in the embedding. An anchor tells CoreSet that the area around a failed pick is covered, so CoreSet picks elsewhere. Anchoring thus filters empty frames without a detector, and it removes nothing from the pool. On iWildCam and Serengeti the gain is fewer failed picks: 238 against 773, and 421 against 724 (Appendix V.3, p. 53). Per labelable frame opened, the skip arm finds slightly more rare classes there (3.9% and 5.0%). Idaho with DINOv3 is the exception: the anchored arm also leads per labelable frame (+43.3%).

When to anchor. Anchor by default. In 58 paired comparisons anchoring helped in 40, made no difference in 12 and hurt in 6, all six on iWildCam after a MegaDetector filter. A lab that always anchors loses 0.94 discovery AUC on average against the better choice in each comparison; a lab that always skips loses 8.24 (Table V.5, p. 61). The revision also tests a warning sign that a running loop can compute: how often the opened neighbors of a failed pick also failed. In a retrospective test it flags 5 of the 6 losses, but a loop that switches on it loses more (1.98) than a loop that always anchors (0.99 on the same 55 comparisons), so the default stays (Appendix V.7, p. 58).

What changed in the PDF. The preregistered part is as reviewed: same design, same hypotheses, same verdicts, and Table 1 is identical. H2 is confirmed, H1 and H3 are not confirmed. The PDF grew from 48 to 67 pages, and nearly all of it answers the reviews: Sec. 4.5 (p. 7) in the main text and Appendix V (pp. 49 to 63). Main-text additions since the reviewed version are in blue. Appendix V is new throughout and opens with a table that maps each review question to the subsection and page that answers it. This new material is exploratory and the paper says so. The first listed contribution is now the empty-frame filter reading above, and the abstract reports the 58 comparisons. The learned-filter result left the abstract and stays in Sec. 4.2 (p. 5). One exception to exploratory: the selector benchmark in Appendix V.8 (p. 61) was registered after the confirmatory runs, and part of it uses earlier runs. It failed both of its tests.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.