acceptodds
Withdrawn from ICLR 2027

When Complete Becomes Partial: Multi-Label Active Learning under Label Expansion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-label active learning aims to train accurate classifiers with less labeling effort by selectively querying instances that might be associated with multiple labels. However, previous studies often assume a fixed set of target categories, without considering the introduction of new categories during annotation. When new categories are introduced, annotations that were complete under the previous label set become partial, leaving unverified labels on historical instances. Therefore, learning under this expansion requires deciding both which unqueried instances to annotate and which historical instances to revisit within a limited query budget. To this end, we introduce Retrospective Multi-Label Active Learning (ReMuAL), a framework that combines first-time queries with selective revisits in a collaborative fashion. Specifically, first-time query feedback updates the model and label relationships, which are then used with verified historical labels to select revisits. Empirically, ReMuAL achieves gains up to 4.67 and 6.45 percentage points in Checklist mAP and full-trajectory AULC respectively over multiple benchmarks.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Active Learning, Multi-label Classification, Label Expansion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dPcGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies a novel setting of multi-label active learning where new classes are introduced during training. They propose how to (i) select which instances to query all currently available labels for, and (ii) determine which previous (already labeled) instances to revisit after a new class has been introduced. Both steps involve standard selection criteria from active learning such as predictive uncertainty and diversity. The unique aspect is that their selection criteria also aims to select instances that are likely to belong to the newly added classes. This is done slightly differently in the two steps, since in (ii) one can utilize the fact that some labels are known for the considered instances.

Critical strengths
  • They propose a novel problem setting for multi-label active learning, which seems practically relevant and has not been studied before.
  • The proposed method is simple and easy to understand, while outperforming baselines in the experiments.
Critical weaknesses
  • The novel aspect of your selection criteria is that you want to prioritize instances that likely contain newly added classes. However, the experiments currently do not evaluate the benefit of this aspect. It is unclear if this component adds any value. If it does not, then the remaining components of your selection criteria are based on ideas that are already well established in the active learning literature.
  • There are a number of issues with the experiments currently which makes it seem like the paper is currently incomplete and not quite ready for publication: (1) on page 18 you say that you evaluate performance on the training set, rather than using a held-out test set for this. This is not standard practice in active learning (or machine learning more broadly) and invalidates all your results in my view. (2) Limited number of baselines, and limited discussion on what these baselines actually do relative your method. (3) Only one seed used for ablations. (4) In Figure 4, your method often has much better performance in round 1, where I assume no class has been introduced. I.e. your method performs clearly better than baselines in a round where the novel aspect of your method is not yet used at all. Why is this? This makes me question the baselines used.
Questions for authors

I don't have any specific questions apart from clarification/discussion of the listed weaknesses above.

Reviewer_QxhbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-label active learning in which the target label set expands over successive learning rounds. After label expansion, previously queried instances retain their existing annotations but lack supervision for newly introduced labels. The authors propose ReMuAL, which coordinates first-time queries and retrospective queries under a shared budget. It first queries new instances to update an intermediate model and estimate label relationships. Model predictions, previously verified positive labels, and the estimated relationships are then used to select historical instances for additional annotation. The final model is trained using all accumulated annotations. Experiments are conducted on several multi-label image datasets with different label-space sizes.

Critical strengths
  1. The paper addresses the practical problem that label expansion makes historical annotations incomplete. Compared with conventional multi-label active learning under a fixed label space, the proposed setting explicitly allocates a shared budget between querying new instances and revisiting historical ones. Although related areas have considered partial-label queries and evolving label spaces separately, their combination in this setting offers a meaningful contribution.

  2. The overall ReMuAL procedure is reasonably aligned with the problem formulation. First-time queries provide initial supervision for newly introduced labels, while the resulting predictions and label relationships guide retrospective queries. This creates a coherent information flow. The experiments cover datasets with substantially different label-space sizes and report performance throughout the learning trajectory.

Critical weaknesses
  1. The current query budget does not accurately represent annotation effort. Every image query is counted as one budget unit, regardless of how many label judgments it contains. Consequently, querying a small number of labels and querying hundreds or even thousands of labels have the same cost. On datasets such as Visual Genome and LVIS, different strategies may therefore receive substantially different amounts of supervision under the same reported budget. This limits the conclusions that can be drawn about annotation efficiency.

  2. The experiments do not sufficiently separate the contribution of retrospective querying from other implementation choices. In addition to its acquisition strategy, ReMuAL introduces label-relation scoring, dynamic budget allocation, specialized label weights, and positive-class rebalancing, while some baselines use different training configurations. Moreover, active selection and evaluation are performed on the same fixed image pool, making the evaluation largely transductive. The reported results demonstrate the effectiveness of the complete system, but they do not clearly identify the source of the gains or establish generalization to an independent test set.

Questions for authors
  1. Figure 2 mainly characterizes conventional MLAL as full-label instance querying under a fixed label space. However, recent work has also considered label-relation modeling and fine-grained image–label queries. Could the authors clarify the key differences between ReMuAL and EMM (NeurIPS 2024) or BESRA(AAAI 2024? If feasible, a comparison with at least one of these methods adapted to the same label-expansion protocol would strengthen the evaluation. If such adaptation is infeasible, please explain the specific obstacles.

  2. Does ReMuAL retain its main advantages when the budget is measured by the number of instance–label judgments rather than image-query calls? At a minimum, reporting the total number of label judgments acquired by each method would help readers assess their actual annotation costs.

  3. ReMuAL uses specialized label weights and positive-class rebalancing in addition to its acquisition strategy. Could the authors provide suitable ablations to clarify the respective contributions of retrospective querying, label-relation modeling, budget allocation, and these training components?

  4. Could the authors further explain why the same fixed image pool is used for active selection and evaluation? If this design represents a particular transductive application, the corresponding assumptions should be stated clearly. If possible, results on an independent test set would help assess generalization.

Reviewer_KEhtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Motivated by the limitation that traditional MLAL assumes a fixed label set, this paper investigates the setting in which label expansion causes previously complete annotations to become partial. It proposes ReMuAL, which coordinates first-time queries and revisits to historical samples under a shared annotation budget. Specifically, ReMuAL first queries new images and updates an intermediate model together with the label relationships, and then selects historical samples for revisiting based on the intermediate predictions, previously verified labels, and the learned label relationships before further updating the model. Experiments on multiple benchmarks show that ReMuAL outperforms state-of-the-art methods in terms of both the final Checklist mAP and the full-trajectory AULC over multiple benchmarks.

Critical strengths
  1. The query criteria address two key challenges in the label-expansion setting: rare newly introduced classes and missing historical labels. For first-time queries, ReMuAL assigns higher weights to potentially rare positive classes and jointly considers label-coverage gains and feature distances during selection, thereby reducing redundant queries of highly similar samples. For historical revisits, it integrates predictions from the current model, inter-label relationships, and previously verified labels to identify samples that are likely to contain newly introduced labels while being supported by historical evidence. The proposed ReMuAL is well motivated and reasonably designed.
  2. The paper evaluates performance throughout the entire label-expansion process rather than focusing only on the final round. This evaluation protocol is well aligned with the continual nature of label expansion, where performance should be assessed as new classes are progressively introduced. Specifically, the paper reports Checklist mAP over all labels released up to each active learning round, evaluates New mAP on newly released label blocks at the corresponding expansion rounds, and uses cumulative AULC to summarize Checklist mAP over the observed trajectory. Full-trajectory AULC is further reported after the complete expansion process. The inclusion of round-wise Checklist mAP and cumulative AULC curves also provides a more informative view of how performance evolves over time, making the empirical evaluation more comprehensive than reporting only endpoint results.
  3. The experimental results provide consistent evidence for ReMuAL’s advantage in cumulative performance over the full label-expansion trajectory. On all datasets, ReMuAL achieves the best results in full-trajectory AULC, which suggest that the main strength of ReMuAL lies in maintaining stronger cumulative performance throughout the progressive label-release process.
Critical weaknesses
  1. The use of \rho_t to determine the revisit budget seems to be a heuristic design choice, implicitly assuming that a higher degree of annotation incompleteness generally warrants allocating more budget to historical revisits. However, \rho_t does not account for the prevalence of the newly introduced classes or how their positive instances are distributed between unqueried samples and historical samples. When a newly introduced class is extremely rare, a higher incompleteness ratio does not imply a higher expected return from revisiting historical samples. Likewise, if positive instances of the new classes are concentrated primarily among samples that have not yet been queried, allocating more budget to first-time queries may be more beneficial than increasing the revisit budget.
  2. The new query strategy introduced by ReMuAL for the label-expansion setting may provide useful insights for active learning under dynamically evolving label spaces. However, the proposed query mechanism is tightly coupled with several other design choices, e.g., positive rebalancing, label weighting, and logit matching for previously observed labels. As a result, it is currently difficult to determine how much of the overall performance gain can be attributed specifically to the proposed query strategy itself.
  3. The practical motivation for the proposed setting could be strengthened. While the paper argues that label expansion may render historical annotations incomplete, it is unclear in which real-world applications revisiting previously annotated samples is actually necessary, and what practical consequences would arise if the newly introduced labels were simply left missing for historical data.
  4. In addition, the current setup relies on a fixed data pool, and assumes that the model predicts over the cumulatively observed label set S_t. These assumptions further limits the practical scope of the proposed method. As a result, it is unclear how well ReMuAL would extend to online active learning, or open-set recognition.
Questions for authors
  1. Could the authors provide a real-world case study demonstrating when retrospective revisiting is necessary and what practical benefits it brings?
  2. How well can ReMuAL generalize beyond the fixed-pool and cumulative-label-set assumptions, particularly to online active learning or open-set settings?
  3. Please refer to Critical Weaknesses.

Comments on the paper

2 comments
Area_Chair_1jzROpenReview ↗

Encouraging Reviewer Feedback on Author Responses

Dear Reviewers,

Thank you all for your time and effort in reviewing and responding. Just a gentle reminder to reviewers to take a look at the authors' responses and share any updated thoughts on whether your main concerns have been addressed and, if not, to indicate which points are not well addressed. The author–reviewer discussion period ends on November 18 (AoE), so I would greatly appreciate your input before then. Thank you again for your time and contributions!

Kind Regards,

AC

Thank you for the response. I am fine with your discussion on the points: Evaluation on the fixed image pool, Baseline coverage and implementation.

Regarding, Part 1: Contribution of new-label-directed acquisition, Part 4: Repeatability of the ablation results, Part 5: First-round performance and common-history controls. Your discussion on these points seem to further confirm to me that the novel aspect of your method does not provide a benefit in terms of performance. Moreover, this is only tested on one seed.

I will therefore keep my score. Please let me know if I misunderstood something.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.