- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies a novel setting of multi-label active learning where new classes are introduced during training. They propose how to (i) select which instances to query all currently available labels for, and (ii) determine which previous (already labeled) instances to revisit after a new class has been introduced. Both steps involve standard selection criteria from active learning such as predictive uncertainty and diversity. The unique aspect is that their selection criteria also aims to select instances that are likely to belong to the newly added classes. This is done slightly differently in the two steps, since in (ii) one can utilize the fact that some labels are known for the considered instances.
Critical strengths
- They propose a novel problem setting for multi-label active learning, which seems practically relevant and has not been studied before.
- The proposed method is simple and easy to understand, while outperforming baselines in the experiments.
Critical weaknesses
- The novel aspect of your selection criteria is that you want to prioritize instances that likely contain newly added classes. However, the experiments currently do not evaluate the benefit of this aspect. It is unclear if this component adds any value. If it does not, then the remaining components of your selection criteria are based on ideas that are already well established in the active learning literature.
- There are a number of issues with the experiments currently which makes it seem like the paper is currently incomplete and not quite ready for publication: (1) on page 18 you say that you evaluate performance on the training set, rather than using a held-out test set for this. This is not standard practice in active learning (or machine learning more broadly) and invalidates all your results in my view. (2) Limited number of baselines, and limited discussion on what these baselines actually do relative your method. (3) Only one seed used for ablations. (4) In Figure 4, your method often has much better performance in round 1, where I assume no class has been introduced. I.e. your method performs clearly better than baselines in a round where the novel aspect of your method is not yet used at all. Why is this? This makes me question the baselines used.
Questions for authors
I don't have any specific questions apart from clarification/discussion of the listed weaknesses above.
Encouraging Reviewer Feedback on Author Responses
Dear Reviewers,
Thank you all for your time and effort in reviewing and responding. Just a gentle reminder to reviewers to take a look at the authors' responses and share any updated thoughts on whether your main concerns have been addressed and, if not, to indicate which points are not well addressed. The author–reviewer discussion period ends on November 18 (AoE), so I would greatly appreciate your input before then. Thank you again for your time and contributions!
Kind Regards,
AC