- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposed a novel active learning algorithm for imbalance in NLP setting. From my perspective, the contribution of the work can be summarized: 1) considering both diversity and uncertainty principles with sampling wrt imbalanced class; 2) leveraging LLMs to generate new data to boost the performance. For the second point, they used 3 LLMs to judge the generative data, which is not a novel point.
Critical strengths
-
Presentation is clear and complete, which can show the methods and emprical results clearly for reading.
-
The imbalance problem they aim to solve is interesting.
Critical weaknesses
METHODS:
- First of all, the authors claim a generative active learning. But in fact, they just used LLMs to generate more data to boost the performance and did not introduce any generative methods for data selection, which is the core of active learning. And even if they introduced a mechanism for "imbalance-aware", they just sampled more data for minority classes and did not propose any new or novel method for "imbalance-aware". From my perspective, generating more data and judge the data using LLMs are not any novel contribution or insights nowadays. And based on their "imbalance" related method, the whole algorithm is not novel or new. They combine the two methods together and did not introduce any new method or insights. Even though they present well, from the novelty or contribution of active learning algorithm, it's not enough. The more generated data can boost the performance and can not show how much their imbalance-aware method can work from intuition. Briefly, they used a combined method and applied it to an imbalance setting. While they did not design something specifically for imbalance itself.
EXPERIMENTS:
- From the empirical results, Figure 3 shows a minor advantage and can not support whether the gap comes from hybird part or imbalance part in method. Other results like Figure 5, they report the comparison between their method (trained on real data and generated data) and baselines (just trained on real data), I think it's not fair unless the baselines work with IGR. Figure 6 & 7 can show the performance of their methods, but the current results still do not clearly show whether the improvement in the first part comes from simply combining diversity and uncertainty, or from the imbalance-aware treatment itself. For the second part, since LLMs are used to generate additional data, it is also unclear whether the performance comparison is fair.
Questions for authors
I have a question for authors. Why they did not report balanced accuracy like DIRECT paper. It's a common metric in imbalance setting.
2 comments
Dear Authors,
Thank you for addressing my comments. I appreciate the additional clarifications and experimental evidence. However, I still have concerns about the methodological novelty. The proposed framework appears to primarily combine existing LLM-based augmentation and active learning techniques, without introducing a fundamentally new selection mechanism. Moreover, although the additional ablations demonstrate the contribution of individual components, I remain unconvinced that the improvements can be specifically attributed to the imbalance-aware design rather than the combination of existing sampling techniques.
Overall, I appreciate the authors' efforts, but these concerns prevent me from substantially changing my assessment of the paper.
We thank the reviewer for the comments.
W1. Using data augmentation to strengthen active learning is well established in computer vision, where cheap perturbations (noise, crops, mixing) are label-preserving. The same recipe largely fails in NLP, where token-level perturbations break semantics (Secs. 1–2). Our contribution is to make this augmentation-enhanced AL paradigm work for NLP through LLM generation, with a refinement stage that keeps synthetic data label-consistent and diverse, and with generation coupled to the evolving labelled pool.
On "generation is not used for selection": synthetic instances already carry their labels, so by construction they are not candidates for annotation; their role is to be learned by the active learner. This acts as a form of knowledge transfer from the LLM into the discriminative model, which strengthens the model that scores the pool and thereby improves selection itself; Table 6 (setting A vs. B) isolates exactly this effect.
On imbalance: GenAL handles it on both sides. IGR synthesizes a class-balanced batch every round (Sec. 3.2), which directly offsets label scarcity in minority classes; it is a plug-and-play module that can be attached to any query strategy (App. C.7), and it outperforms other augmentation methods on the imbalanced benchmarks (Fig. 14). BIRDS builds its balanced subpool from ensemble pseudo-labels with an explicit hard-to-learn class and no label priors (Eq. 8–9), and its queried set has the lowest or among the lowest MeanIR across rounds (Fig. 18).
W2a. The margins in Fig. 3 are tested in Fig. 8 (App. C.1): a pairwise significance matrix from paired t-tests over the five-seed distributions, following the protocol of Tan et al. (2023). After Holm correction (revised Fig. 8), GenAL retains 19 of 20 pairwise wins on the standard benchmarks and 17 of 20 on the imbalanced ones, the most of any strategy in both regimes. Whether the gain comes from combining diversity and uncertainty or from the imbalance-aware treatment is separated by the ablation of the three BIRDS scoring components in Sec. 4.6 and App. C.6 (Fig. 22): removing pseudo-label-guided subsampling alone causes a clear drop, distinct from removing diversity or uncertainty. We keep this ablation in the BIRDS-specific analysis (Sec. 4.6) because it dissects the scoring terms inside BIRDS, whereas Sec. 4.5 compares the two modules of GenAL.
W2b. Fig. 5 is not the main comparison. Its purpose is to show how GenAL compares with baselines without IGR when labelled data is very scarce (half the human budget), i.e., how much synthetic data helps in that regime (Sec. 4.4). The controlled comparisons are Fig. 3, where every baseline query strategy is equipped with the same IGR module and the same synthetic budget, and Figs. 6, 11 and 13, where no method uses IGR (BIRDS alone against the baseline strategies); GenAL / BIRDS leads in both. Likewise, Figs. 7, 12 and 14 compare IGR against other augmentation methods under the same query strategy.
Q. We report Macro-F1 on all imbalanced settings. Balanced accuracy is the mean of per-class recall; Macro-F1 is the mean of per-class F1, i.e., the harmonic mean of precision and recall for each class. Both give every class equal weight regardless of its frequency, which is the property that matters under imbalance, and Macro-F1 is the stricter of the two because it also penalises false positives on minority classes. It is the standard metric for imbalanced text classification (e.g., Schröder & Heyer, 2024 [1]; Ghose & Nguyen, 2024 [2]; Lesci & Vlachos, 2024 [3]), so we keep it as the primary metric.
[1] Schröder, C., & Heyer, G. Self-training for sample-efficient active learning for text classification with pre-trained language models. EMNLP 2024.
[2] Ghose, A., & Nguyen, E. T. On the fragility of active learners for text classification. EMNLP 2024.
[3] Lesci, P., & Vlachos, A. AnchorAL: Computationally efficient active learning for large and imbalanced datasets. NAACL 2024.