acceptodds
Under review as a conference paper at ICLR 2027

GenAL: Imbalance-aware Generative Active Learning for Discriminative Models

Abstract

Standard active learning (AL) frameworks for discriminative models struggle with imbalanced datasets due to misjudgment of sample informativeness under class imbalance. Current methods typically rely on algorithmic or sampling-based techniques to address imbalanced distributions, however, they overlook a more intrinsic solution: generative data augmentation. While traditional data augmentation approaches often fail in NLP tasks due to semantic loss, Large Language Models, with remarkable generalization and data synthesis capabilities, provide a promising pathway to mitigate class imbalance and further improve AL performance. In this paper, we introduce GenAL, a generative active learning framework robust across both standard and imbalanced datasets. To specifically address class imbalance, we design an imbalance-aware query strategy (BIRDS), coupled with a generative annotation framework (IGR) incorporating an automated refinement mechanism to synthesize diverse, high-quality data for AL training. Extensive experiments across both standard and imbalanced settings demonstrate that GenAL and BIRDS achieve superior performance compared to state-of-the-art query strategies, while IGR consistently delivers performance gains across different baselines.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Active Learning, LLM-in-th-Loop

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_xfteOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed a novel active learning algorithm for imbalance in NLP setting. From my perspective, the contribution of the work can be summarized: 1) considering both diversity and uncertainty principles with sampling wrt imbalanced class; 2) leveraging LLMs to generate new data to boost the performance. For the second point, they used 3 LLMs to judge the generative data, which is not a novel point.

Critical strengths
  1. Presentation is clear and complete, which can show the methods and emprical results clearly for reading.

  2. The imbalance problem they aim to solve is interesting.

Critical weaknesses

METHODS:

  1. First of all, the authors claim a generative active learning. But in fact, they just used LLMs to generate more data to boost the performance and did not introduce any generative methods for data selection, which is the core of active learning. And even if they introduced a mechanism for "imbalance-aware", they just sampled more data for minority classes and did not propose any new or novel method for "imbalance-aware". From my perspective, generating more data and judge the data using LLMs are not any novel contribution or insights nowadays. And based on their "imbalance" related method, the whole algorithm is not novel or new. They combine the two methods together and did not introduce any new method or insights. Even though they present well, from the novelty or contribution of active learning algorithm, it's not enough. The more generated data can boost the performance and can not show how much their imbalance-aware method can work from intuition. Briefly, they used a combined method and applied it to an imbalance setting. While they did not design something specifically for imbalance itself.

EXPERIMENTS:

  1. From the empirical results, Figure 3 shows a minor advantage and can not support whether the gap comes from hybird part or imbalance part in method. Other results like Figure 5, they report the comparison between their method (trained on real data and generated data) and baselines (just trained on real data), I think it's not fair unless the baselines work with IGR. Figure 6 & 7 can show the performance of their methods, but the current results still do not clearly show whether the improvement in the first part comes from simply combining diversity and uncertainty, or from the imbalance-aware treatment itself. For the second part, since LLMs are used to generate additional data, it is also unclear whether the performance comparison is fair.
Questions for authors

I have a question for authors. Why they did not report balanced accuracy like DIRECT paper. It's a common metric in imbalance setting.

2 comments

We thank the reviewer for the comments.

W1. Using data augmentation to strengthen active learning is well established in computer vision, where cheap perturbations (noise, crops, mixing) are label-preserving. The same recipe largely fails in NLP, where token-level perturbations break semantics (Secs. 1–2). Our contribution is to make this augmentation-enhanced AL paradigm work for NLP through LLM generation, with a refinement stage that keeps synthetic data label-consistent and diverse, and with generation coupled to the evolving labelled pool.

On "generation is not used for selection": synthetic instances already carry their labels, so by construction they are not candidates for annotation; their role is to be learned by the active learner. This acts as a form of knowledge transfer from the LLM into the discriminative model, which strengthens the model that scores the pool and thereby improves selection itself; Table 6 (setting A vs. B) isolates exactly this effect.

On imbalance: GenAL handles it on both sides. IGR synthesizes a class-balanced batch every round (Sec. 3.2), which directly offsets label scarcity in minority classes; it is a plug-and-play module that can be attached to any query strategy (App. C.7), and it outperforms other augmentation methods on the imbalanced benchmarks (Fig. 14). BIRDS builds its balanced subpool from ensemble pseudo-labels with an explicit hard-to-learn class and no label priors (Eq. 8–9), and its queried set has the lowest or among the lowest MeanIR across rounds (Fig. 18).

W2a. The margins in Fig. 3 are tested in Fig. 8 (App. C.1): a pairwise significance matrix from paired t-tests over the five-seed distributions, following the protocol of Tan et al. (2023). After Holm correction (revised Fig. 8), GenAL retains 19 of 20 pairwise wins on the standard benchmarks and 17 of 20 on the imbalanced ones, the most of any strategy in both regimes. Whether the gain comes from combining diversity and uncertainty or from the imbalance-aware treatment is separated by the ablation of the three BIRDS scoring components in Sec. 4.6 and App. C.6 (Fig. 22): removing pseudo-label-guided subsampling alone causes a clear drop, distinct from removing diversity or uncertainty. We keep this ablation in the BIRDS-specific analysis (Sec. 4.6) because it dissects the scoring terms inside BIRDS, whereas Sec. 4.5 compares the two modules of GenAL.

W2b. Fig. 5 is not the main comparison. Its purpose is to show how GenAL compares with baselines without IGR when labelled data is very scarce (half the human budget), i.e., how much synthetic data helps in that regime (Sec. 4.4). The controlled comparisons are Fig. 3, where every baseline query strategy is equipped with the same IGR module and the same synthetic budget, and Figs. 6, 11 and 13, where no method uses IGR (BIRDS alone against the baseline strategies); GenAL / BIRDS leads in both. Likewise, Figs. 7, 12 and 14 compare IGR against other augmentation methods under the same query strategy.

Q. We report Macro-F1 on all imbalanced settings. Balanced accuracy is the mean of per-class recall; Macro-F1 is the mean of per-class F1, i.e., the harmonic mean of precision and recall for each class. Both give every class equal weight regardless of its frequency, which is the property that matters under imbalance, and Macro-F1 is the stricter of the two because it also penalises false positives on minority classes. It is the standard metric for imbalanced text classification (e.g., Schröder & Heyer, 2024 [1]; Ghose & Nguyen, 2024 [2]; Lesci & Vlachos, 2024 [3]), so we keep it as the primary metric.

[1] Schröder, C., & Heyer, G. Self-training for sample-efficient active learning for text classification with pre-trained language models. EMNLP 2024.

[2] Ghose, A., & Nguyen, E. T. On the fragility of active learners for text classification. EMNLP 2024.

[3] Lesci, P., & Vlachos, A. AnchorAL: Computationally efficient active learning for large and imbalanced datasets. NAACL 2024.

Dear Authors,

Thank you for addressing my comments. I appreciate the additional clarifications and experimental evidence. However, I still have concerns about the methodological novelty. The proposed framework appears to primarily combine existing LLM-based augmentation and active learning techniques, without introducing a fundamentally new selection mechanism. Moreover, although the additional ablations demonstrate the contribution of individual components, I remain unconvinced that the improvements can be specifically attributed to the imbalance-aware design rather than the combination of existing sampling techniques.

Overall, I appreciate the authors' efforts, but these concerns prevent me from substantially changing my assessment of the paper.

Reviewer_Dd4hOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a generative active learning framework for text classification tasks. It is composed of two stages, first (BIRDS) is the diversity and uncertainty score-based active learning selection module, and the second is the generative data augmentation part based on LLM. The paper's contribution lies in combining uncertainty + diversity-based sampling methods into selection part, and buliding a robust text synthesis pipeline. Evaluation on four famous text classification benchmarks outperform existing baseline methods.

Critical strengths

Various ideas in previous active learning and synthesis literatures are put together to solve the generative active learning under imbalance. The extensive experiments and ablations on their modules show that they are built concretely as a generative active learning framework.

Critical weaknesses

Most parts of the methods are a composition of methods of previous methods, some parts misses critical citations. For example, majority-vote of LLMs are widely used such as “Panel of LLM evaluators” [1], and subtopic-conditioned generation also is identical to that of AttrPrompt [2]. Also, be more precise on claiming where your contribution lies at and which are brought from existing works.

[1] Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., ... & Lewis, P. (2024). Replacing judges with juries: Evaluating llm generations with a panel of diverse models. arXiv preprint arXiv:2404.18796.

[2] Yu, Y., Zhuang, Y., Zhang, J., Meng, Y., Ratner, A. J., Krishna, R., ... & Zhang, C. (2023). Large language model as attributed training data generator: A tale of diversity and bias. Advances in neural information processing systems, 36, 55734-55784.

Questions for authors
  1. In the comparisons of Figure 3, do the IGR-augmented baselines regenerate synthetic data from their own updated labeled pool each round (Table 6 setting A), or from the initial seed set (setting C)? If the former, what justifies attributing the gap to “adaptive coupling” rather than to the BIRDS alone?

  2. How does IGR relate to AttrPrompt [2] and the Panel-of-LLM-Evaluators approach [1]? Is there a difference beyond integration into the AL loop?

[1] Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., ... & Lewis, P. (2024). Replacing judges with juries: Evaluating llm generations with a panel of diverse models. arXiv preprint arXiv:2404.18796.

[2] Yu, Y., Zhuang, Y., Zhang, J., Meng, Y., Ratner, A. J., Krishna, R., ... & Zhang, C. (2023). Large language model as attributed training data generator: A tale of diversity and bias. Advances in neural information processing systems, 36, 55734-55784.

4 comments

We thank the reviewer for reading the appendix closely and for the constructive suggestions.

Q1. Yes: all curves in Fig. 3 are run under setting A. We understand the reviewer's point about what this comparison can show. Since every method shares the same IGR module and synthetic budget and regenerates from its own newly annotated data each round, Fig. 3 compares fully coupled systems and shows that the BIRDS–IGR pairing is the most effective one. BIRDS contributes through two channels: directly, through the examples it sends for annotation, and indirectly, because those examples become the reference data and subtopic source for IGR in the same round (Eq. 10). This second channel is what "adaptive coupling" in Finding 1 refers to; since the phrase can also be read as a claim about coupling as a mechanism, Finding 1 is reworded in the revised version to make this explicit. Coupling as a mechanism is tested separately in Table 6 / Fig. 15, where BIRDS and IGR are both present and only the information flow between them is removed.

Q2. We thank the reviewer for these two references; we overlooked both and have added them in the revised version. AttrPrompt (Yu et al., 2023) is close to IGR in the idea of conditioning generation on subtopics to avoid mode collapse. The difference lies in how the subtopics are obtained: AttrPrompt lets the LLM propose attribute dimensions and candidate values once, in a zero-shot setting, with a human-in-the-loop step to select the candidates, whereas IGR has no human step and derives the subtopics automatically at every AL round from the newly annotated examples (Eq. 10), uses those examples as few-shot context, and filters the generated instances afterwards rather than the attribute values beforehand. For the judge, the multi-model majority vote now cites Verga et al. (2024).

Both citations, with a short note on the relation to these works, are now in the related work (Sec. 2) and in Sec. 3.2, marked in blue.

Thank you for the clarifications and the revision. The missing references are now cited and Finding 1 is fixed, which resolves my Q2 and most of Q1. I have also read your responses to the other reviewers. Two observations remain, so please correct me if I have misread anything.

  1. Coupling. The novelty of the framework is defended mainly through the adaptive coupling (Table 6 / Fig. 15). In the revised Fig. 15, however, the four settings appear nearly indistinguishable, especially on PubMed and DBpedia, and unlike Fig. 8 no significance test is reported. Is it correct that the measured effect of coupling is small? If so, the statements in the contribution list, Sec. 4.5 and App. C.2 seem stronger than the evidence.

  2. Components and imbalance. My reading is that the new elements are the pseudo-label-guided subsampling in BIRDS and the automatic, per-round subtopic derivation in IGR, while the rest follows BEMPS, AttrPrompt and Verga et al. On imbalance, the mechanisms appear to be an equal per-class generation quota and a subpool balanced by predicted class with an added hard-to-learn group; both seem to be simple adaptations of standard practice, with limited novelty. In addition, the subsampling ablation (Fig. 22) is reported only on the standard benchmarks, so its contribution under imbalance is not shown clearly. Is this reading correct?

Thank you for the follow-up. One clarification first, since it bears on both points: the novelty of GenAL does not rest mainly on the coupling. The paper proposes a framework that brings augmentation into active learning for NLP to address class imbalance. This is common practice in vision but remains under-explored in NLP, where token-level augmentation breaks semantics and LLM generation has to stay label-consistent, diverse and class-balanced inside the AL loop (Secs. 1–2). Within this framework, each component also outperforms its counterparts on its own: BIRDS against existing query strategies without IGR (Figs. 6, 11, 13), and IGR against other augmentation methods (Figs. 7, 12, 14).

1. Fig. 15 is the ablation of the coupling design, and its gain depends on the dataset. On DBpedia the four settings coincide because the task is close to saturation: all strategies reach 98–99% at the final round, which leaves little headroom for any design choice. On the other three datasets the ordering at the final round is the same: the fully coupled setting A is the best, removing one direction of information flow (B or C) lowers the result, and removing both (D) gives the lowest. The margin of A over the best decoupled variant is 0.9 points on SST-2, 0.5 on AGNews and 0.3 on PubMed (table below). Coupling therefore adds a further gain on top of the two components; it is moderate in size, consistent in direction, and varies across datasets.

Setting (Fig. 15)SST-2AGNewsPubMedDBpedia
A (GenAL)85.97 ± 0.6289.97 ± 0.3279.09 ± 0.5198.71 ± 0.09
B (frozen BIRDS)84.90 ± 0.8189.49 ± 0.3278.77 ± 0.2298.66 ± 0.10
C (frozen IGR)85.09 ± 0.9789.33 ± 0.2878.80 ± 0.1098.66 ± 0.10
D (fully decoupled)84.44 ± 1.4789.25 ± 0.4978.55 ± 0.3098.69 ± 0.17

Micro-F1 after round 10, mean ± std over five seeds, values in %.

2. Fig. 18 already reports how BIRDS behaves on the imbalanced datasets. It plots the MeanIR of the queried labelled set at each round, i.e., how imbalanced the selected data is in label space (lower is more balanced). BIRDS has the lowest or among the lowest MeanIR across rounds, and the lowest at the final round at all four imbalance levels; querying more balanced data is one reason for its performance under imbalance. In addition, we ran the subsampling ablation on the imbalanced PubMed variants (figure): the full BIRDS is higher than the no-subsampling variant at all four imbalance levels, which shows that the subsampling is effective under imbalance.

Thank you for the further response and the additional results. The discussion has clarified two points that weigh on my assessment. First, the benchmarks do not show the effect of coupling clearly: on DBpedia all strategies reach 98–99% at the final round, and on the other datasets the margins are small, so the effect is difficult to verify. Second, the elements specific to this work are limited to the pseudo-label-guided subsampling and the per-round subtopic derivation, which seem effective but are simple adaptations of existing techniques. I value the careful evaluation, but given the limited novelty of the components and the modest gains over the strongest baselines, I am lowering my rating to weak reject.

Reviewer_1fgeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

GenAL combines imbalance-aware active learning with LLM-based data generation. BIRDS scores candidates using feature-space diversity and an MC-dropout approximation of proper-score change, forms a pseudo-label-balanced subpool, and selects queries through K-Means. IGR generates class-conditioned synthetic text from newly labeled examples and filters it using majority agreement among three LLM judges. Experiments use SST-2, AGNews, PubMed, DBpedia, and controlled imbalanced variants with DistilBERT and DeBERTa backbones. The paper reports improved learning curves over several acquisition and augmentation baselines and includes analyses of coupling, generation budgets, thresholds, judge protocols, model scale, and runtime.

Critical strengths
  • The paper jointly addresses annotation budgets, class imbalance, and the quality of generated data, which is a practically important problem.
  • The evaluation covers standard and imbalanced settings, two classification backbones, and five random seeds.
  • The paper attempts to place human annotation and synthetic-data generation under a unified budget and includes a half-budget experiment.
  • The appendix includes analyses of coupling, components, thresholds, temperature, judge protocols, model scale, and runtime.
  • The full prompt templates are provided, and the experiments primarily use local 7B-9B models.
Critical weaknesses

Here I have some majpr concerns:

  1. The dataset protocol is internally inconsistent. The authors must explain the source of DBpedia's MeanIR=4.6 and the SST-2 test labels for the 1,821 examples, and provide the exact Hugging Face dataset ID, revision, split, and per-class counts.
  2. BIRDS is underspecified at several decision-critical points. Equation (3) defines S_U(x) as a vector over randomly selected anchors, but the paper does not report the number of anchors, their sampling procedure, or whether they are resampled across rounds. Equation (6) then concatenates the scalar diversity score S_D(x) with the multidimensional S_U(x) after "min-max normalization," without specifying whether normalization is performed per dimension or globally, or how the two components are weighted. Because Euclidean K-Means is applied in this joint space, the number and scaling of the anchor dimensions can substantially change the selected query batch. The actual proper scoring rule, quota rounding and redistribution when a pseudo-class has insufficient candidates, and K-Means initialization are also unspecified. These choices are algorithmically consequential rather than cosmetic implementation details.
  3. Algorithm 1 conflicts with the pool definitions used in the coupling ablation. Line 15 updates L by adding both newly human-labeled examples and D_t. If the next IGR round uses L_t as in Equation (10), previous synthetic examples become generation context, creating a recursive synthetic-to-synthetic feedback loop. In contrast, Table 6 explicitly defines L_t as the cumulative human-labeled pool and D_t as a separate synthetic pool: IGR conditions only on L_t, while the BIRDS model is trained on L_t union D_t. These are materially different algorithms with different error-propagation behavior and different interpretations of the claimed coupling. It is also unclear whether D_t is round-specific or cumulative; if it is cumulative, Algorithm 1 may repeatedly add old synthetic examples. The paper should state which protocol was used in code and provide corrected pseudocode with separate human and synthetic pools.
  4. The primary statistical evidence is insufficient. The paper reports only range bands and win counts from multiple paired t-tests with n=5, without effect sizes, raw p-values, confidence intervals, or corrections for multiple comparisons.
  5. Several claims go beyond the evidence in the figures and tables. Figure 3 cannot identify the effect of coupling; Tables 3 and 7 do not show consistent improvements for every model and metric; and Figure 15 covers only two datasets and contains crossing curves.

and I also have below comments:

  • The paper does not explain how the initial 50 labeled examples are sampled. Balanced initialization would use knowledge of the true labels, while random initialization may omit some classes in 14-class or high-MeanIR settings, preventing IGR from generating examples for them.
  • Figure 19 excludes hard-to-learn examples and reports only aggregate pseudo-label accuracy. Minority-class precision, recall, and coverage under high MeanIR are missing.
  • Simple but important controls are absent, including class-balanced sampling, oversampling/reweighting, and BEMPS + the same balanced subpool.
  • The LLM judge is validated only using randomly corrupted labels, whose error distribution differs from that of genuine generation errors. The validation-set size is not reported.
  • Human evaluation covers only 200 examples and is performed by three authors, without blinding, a detailed rubric, or confidence intervals.
  • The IGR formulation states that L_k^t is used as few-shot context, but the prompt template appears to contain only one example. The actual example-selection and truncation procedures are unclear.
  • The reported 2-4 minute runtime for four LLMs on a single RTX 3090 lacks essential details, including, quantization, batching.
Questions for authors
  1. Please provide the dataset identifier, revision, split, per-class counts, and sample manifest, and explain the DBpedia and SST-2 discrepancies.
    • If these discrepancies cannot be reconciled, they constitute grounds for rejection. If they are merely table errors and the protocol is verifiable, I could reassess the paper.
  2. Please report exact means and standard deviations over the five seeds for every dataset, round, and method, together with paired final-round differences, 95% confidence intervals, and corrections for multiple comparisons.
    • This evidence is necessary to determine whether the reported improvements are genuinely stable.
  3. In the actual implementation, does L_t contain only human-labeled data? Is historical synthetic data included in the context for subsequent IGR rounds?
    • Algorithm 1 or Table 6 should be corrected accordingly, and the recursive-generation policy should be stated explicitly.
  4. How many anchors does BIRDS use, which proper score is used, how are the components normalized, and how are quotas handled?
    • Without these details, the method's complexity and fairness cannot be assessed.
  5. Can you add BEMPS + the same balanced subpool and simple class-balanced sampling baselines?
    • A consistent advantage over these controls is needed to attribute the gains to BIRDS's more elaborate uncertainty design.
  6. Is the initial seed set sampled randomly or stratified? Under MeanIR 20/50, what are the per-class precision, recall, and coverage of the pseudo-labels?
    • This determines whether the method genuinely addresses minority classes rather than amplifying pseudo-label bias.
4 comments

We thank the reviewer for the thorough and precise review.

W1 / Q1. All datasets are loaded from the Hugging Face without modification: AGNews from fancyzhx/ag_news, PubMed from pietrolesci/pubmed-20k-rct, DBpedia from fancyzhx/dbpedia_14, and SST-2 with the training split from stanfordnlp/sst2 and the test split from SetFit/sst2.

On the DBpedia discrepancy, the reviewer is right. dbpedia_14 is class-balanced. The 4.6 in Table 1 came from our dataset-statistics script, which counted characters of the label string rather than labels, so the two-digit labels 10–13 were split; this reproduces exactly 4.6, and the other three datasets are unaffected because their labels are single digits. The bug is confined to that statistic; training, acquisition and the imbalanced variants of Table 5 (Algorithm 2) all use the integer label field. No experiment is affected. We have corrected Table 1 in the revised version; thank you for pointing this out.

W2 / Q4. We answer each point below; the corresponding details are now stated in the revised Secs. 3.1 and 4.1 (marked in blue).

  1. Anchors. 1% of the current unlabelled pool, drawn uniformly at random and resampled every round (now stated in Sec. 4.1).
  2. Proper scoring rule. The Brier (MSE) score.
  3. Normalization. Anchor dimensions with a score range below 10−510^{-5} across candidates are removed (anchor filtering, Sec. 3.1). Min-max normalization is then applied per anchor dimension, and to the diversity score, over the candidates, so all coordinates of S(x)\mathcal{S}(x) share the same range (now stated in Sec. 3.1).
  4. Weighting of the two components. The blocks are concatenated without reweighting, by design: the uncertainty vector carries more information than the scalar diversity score, which enters as a complementary coordinate rather than an equally weighted objective. Fig. 22 (no-diversity) shows that this coordinate still matters. We also ran an ablation that scales the uncertainty block by 1/d1/\sqrt{d} before concatenation, so that the two scores become equally important (figure); it does not perform better than the original design.
  5. Quotas. Eq. 9 allocates the subpool budget (now written Bsub=10BB_{\mathrm{sub}} = 10B to distinguish it from BB). Standard pseudo-classes are sampled first, without replacement; the hard-to-learn group is sampled last and takes the remaining budget, so the subpool always has exactly 10B10B candidates and rounding is absorbed there (now stated after Eq. 9).
  6. K-Means. scikit-learn defaults (k-means++ initialization); the instance nearest to each centroid is retained, as described in Sec. 3.1.

W3 / Q3. The inconsistency is a notation clash, not two different algorithms. In Algorithm 1, Lt\mathcal{L}^t and Dt\mathcal{D}^t denote the batches annotated and generated at round tt; Table 6 reused the same symbols for the cumulative pools used to train the BIRDS model, and did not explicitly distinguish the generation context. In the implementation, IGR conditions only on the human examples newly annotated at round tt (Eq. 10–11). Synthetic data therefore never enters the generation context and there is no synthetic-to-synthetic generation; Dt\mathcal{D}^t is round-specific, and each batch is added to the training pool exactly once. In the revised version, Table 6 writes the cumulative pools as L:t\mathcal{L}^{:t} and D:t\mathcal{D}^{:t}, and Algorithm 1 defines Lt\mathcal{L}^t explicitly and updates the human-labelled and synthetic pools separately, as the reviewer suggested. We thank the reviewer for pointing this out.

W4 / Q2. The win counts in Fig. 8 come from paired t-tests over the five seeds, following Tan et al. (2023). As the reviewer suggested, the revised Fig. 8 now applies Holm correction for multiple comparisons within each dataset, over each method's five comparisons against the other methods. After correction, GenAL retains 19 of 20 pairwise wins on the standard benchmarks and 17 of 20 on the imbalanced ones, and remains the strategy with the most wins in both regimes; the caption of Fig. 8 and App. C.1 are updated accordingly. We also provide below the final-round mean ± std over the five seeds for every dataset and method in Fig. 8.

MethodSST-2AGNewsPubMedDBpediaPubMed-5PubMed-10PubMed-20PubMed-50
Random+IGR83.98 ± 1.6088.86 ± 0.4877.96 ± 0.5398.09 ± 0.1865.94 ± 1.2264.62 ± 1.0764.21 ± 0.6763.34 ± 1.02
CoreSet+IGR84.07 ± 1.5889.29 ± 0.3577.66 ± 0.5198.52 ± 0.1467.27 ± 0.9565.92 ± 1.4465.47 ± 1.0465.62 ± 1.99
BEMPS+IGR85.05 ± 0.3589.66 ± 0.4378.65 ± 0.5298.60 ± 0.1666.19 ± 1.4365.44 ± 2.1565.66 ± 1.5665.71 ± 1.42
AnchorAL+IGR83.51 ± 0.5288.91 ± 0.5176.48 ± 0.8398.12 ± 0.4866.01 ± 0.9565.19 ± 0.6464.76 ± 0.7464.21 ± 2.02
DIRECT+IGR84.35 ± 0.7889.11 ± 0.4377.82 ± 0.5298.19 ± 0.1566.72 ± 0.9265.02 ± 0.8265.00 ± 1.0165.10 ± 1.97
GenAL85.97 ± 0.6289.97 ± 0.3279.09 ± 0.5198.71 ± 0.0967.97 ± 0.8466.95 ± 0.3166.29 ± 0.7666.34 ± 1.26

Standard benchmarks: Micro-F1 after round 10 (DistilBERT); imbalanced PubMed variants: Macro-F1 after round 6 (DeBERTa); values in %.

W5. Fig. 3. All curves, baselines included, use the same IGR module and synthetic budget and regenerate from their own newly annotated data each round, so Fig. 3 compares fully coupled systems and shows that the BIRDS–IGR pairing is the most effective one. BIRDS contributes both through the examples it selects and through the generation context those examples provide to IGR (Eq. 10); this is what "adaptive coupling" in Finding 1 refers to. Since the phrase can also be read as a claim about coupling as a mechanism, which is tested separately in Table 6 / Fig. 15, Finding 1 is reworded in the revised version to make this explicit. Tables 3 and 7. The statements are revised to describe the results exactly as reported. Fig. 15. The coupling ablation is extended to all four standard benchmarks in the revised Fig. 15.

Other questions.

  1. Initial seed set. The 50 initial examples are sampled at random, without access to labels (revised in Sec. 4.1). If a class has no labelled example, IGR generates zero-shot for that class until one is annotated (revised in Sec. 3.2). The extreme case, where a minority class stays unlabelled for most rounds, is the cold-start regime stated as a limitation and future work in the conclusion.
  2. Pseudo-label quality on minority classes (Q6). In BIRDS, pseudo-labels are a means to balanced acquisition. They are used only to draw the candidate subpool as evenly as possible across classes, and instances without a confident prediction are not forced into a class but kept in a separate hard-to-learn group (Eq. 8), which limits the propagation of pseudo-label errors. Whether this design genuinely serves minority classes is best judged by what is actually queried, which Fig. 18 reports using gold labels. If the subsampling merely amplified pseudo-label bias toward the majority classes, the gold-label MeanIR of the queried set would stay close to that of the pool. Instead, the MeanIR of the set queried by BIRDS falls well below that of the pool at every imbalance level, including MeanIR 20 and 50, and is the lowest or among the lowest of all strategies across rounds, which indicates that the pseudo-labels and the overall design of BIRDS are effective in handling class imbalance. Fig. 19 complements this with pseudo-label accuracy and coverage as a function of the threshold.
  3. Simple controls (Q5). The no-diversity variant in Fig. 22 serves as this control: it selects by uncertainty alone on the same balanced subpool. The full BIRDS outperforms it on both datasets, so the gain does not come from the balanced subpool alone. Conversely, the no-subsampling variant shows that the subpool itself is necessary. Class-balanced acquisition under imbalance is represented by the established imbalance-aware strategies DIRECT and AnchorAL, which we compare against throughout.
  4. LLM-judge validation. The validation set has 500 examples per dataset (now stated in Sec. 4.7). Randomly corrupted labels do differ from genuine generation errors, but they are the harsher test: a judge that cannot detect label–text mismatch scores at chance, whereas the ensemble reaches 0.80–0.99 (Table 4), which demonstrates its effectiveness. Errors of real generations are assessed directly on generated samples by the human evaluation (Table 8); the two checks are complementary.
  5. Human evaluation. Human evaluation at this scale is common practice (Wang et al., 2023 [1]; Honovich et al., 2023 [2]; Gekhman et al., 2023 [3]). The three annotators labelled independently with a binary good/poor criterion (label consistency and fluency); agreement is reported as Gwet's AC1 (Table 8). The revised App. C.5 states the criterion and the sample size, and the revised Table 8 adds 95% Wilson confidence intervals.
  6. Few-shot context. The {example} field in App. D.1 is a placeholder marking where reference examples are inserted; up to five examples are drawn uniformly at random from the newly annotated examples of class kk at each round, as now stated in Sec. 3.2.
  7. Runtime. All four LLMs run in bfloat16 without quantization, batch size 1, on a single RTX 3090; only one LLM is resident at a time (the generator, then each judge in turn, with the previous model released before the next is loaded). The 2–4 minutes per round in Table 9 include this loading and unloading. These details are now stated in App. C.7.

[1] Self-Instruct: Aligning Language Models with Self-Generated Instructions (Wang et al., ACL 2023)

[2] Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (Honovich et al., ACL 2023)

[3] TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models (Gekhman et al., EMNLP 2023)

Thank you for the detailed responses and revisions. I appreciate the clarifications regarding the dataset statistics, BIRDS implementation, and the separation of human-labeled and synthetic data. These address several of my original concerns.

However, my main concern remains the limited evidence supporting the methodological contribution. In particular, the effectiveness of adaptive coupling is still not sufficiently established, and the current ablations do not fully isolate the benefits of BIRDS's uncertainty design from balanced subsampling. A comparison with BEMPS using the same balanced subpool would be particularly helpful.

While the revisions improve the paper's clarity and reproducibility, I remain unconvinced that the experimental evidence sufficiently supports the claimed contributions. I therefore maintain my rejection recommendation.

Reviewer_M2RROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GenAL, an imbalance-aware generative active learning framework that integrates two main components: Bayesian Imbalance-aware Representative Diversity Sampling (BIRDS) and Ideate-Generate-Refine (IGR). BIRDS combines feature-space diversity, Bayesian uncertainty estimation, and pseudo-label-guided balanced subsampling to select informative samples for human annotation. IGR leverages LLMs to synthesize class-conditioned training instances using dynamically generated subtopics, followed by a multi-LLM majority-voting mechanism for quality filtering. The two components interact iteratively: newly annotated samples guide subsequent data generation, while synthetic data updates the discriminative model used for future acquisition. Experiments on standard and artificially imbalanced text classification benchmarks demonstrate improvements over several active learning and data augmentation baselines.

Critical strengths
  1. The paper addresses a practically important problem. It focuses on class imbalance and labeled data scarcity in active learning and integrates active learning with LLM-based data augmentation, offering potential practical value.
  2. The overall framework is reasonably designed. The proposed GenAL framework integrates imbalance-aware acquisition (BIRDS) with generative data augmentation (IGR), establishing an iterative learning mechanism in which sample selection and data generation mutually benefit each other.
Critical weaknesses
  1. The methodological novelty is relatively limited. BIRDS primarily combines existing techniques, including feature diversity, uncertainty estimation, and class-balanced sampling. Similarly, the subtopic-guided generation and multi-LLM voting mechanisms in IGR are based on commonly adopted techniques. Therefore, the overall contribution appears to rely more on integrating existing methods than on introducing fundamentally new methodological insights.
  2. The fusion of diversity and uncertainty in BIRDS lacks sufficient justification. In Eq. (6), the authors directly concatenate a scalar diversity score with a multidimensional uncertainty vector and employ K-Means for sample selection. However, Min-Max normalization does not eliminate the potential dominance of the higher-dimensional uncertainty representation in distance calculations. Further justification and comparisons with alternative fusion strategies are recommended.
  3. Some experimental results do not fully support the claims made in the manuscript. For example, Table 3 shows that Guided Prompt performs worse than Vanilla Prompt in terms of the Sentence-BERT diversity metric across all four generators on PubMed. However, the authors claim that guided prompting consistently improves diversity across all models. This inconsistency should be clarified, and the corresponding conclusions should be revised.
  4. The evaluation of real-world generalization and computational efficiency remains limited. Most imbalance experiments rely on artificially constructed class distributions, leaving the effectiveness of the proposed approach on naturally occurring long-tailed datasets insufficiently explored.
Questions for authors

See Weaknesses

1 comment

We thank the reviewer for the careful reading and the constructive comments.

W1. We respectfully disagree that the contribution reduces to integrating existing methods. The paper offers a different way of attacking class imbalance in active learning for NLP tasks: augmenting the learner with LLM generation inside the AL loop, a direction that remains under-explored (Sec. 2). Making this work is not a matter of assembling components: a static combination of acquisition and generation does not deliver the gains; the adaptive coupling between them does (Table 6 / Fig. 15), and the acquisition side has to remain reliable under imbalance.

W2. The uncertainty block does have the larger influence on the clustering, and this is intended: the uncertainty vector carries more information than the scalar diversity score, which enters as a complementary signal rather than an equally weighted objective. Fig. 22 shows that the diversity score still matters: removing it (no-diversity) lowers performance.

As suggested, we also ran an ablation that scales the uncertainty block by 1/d1/\sqrt{d} before concatenation, so that the two scores become equally important (figure); it does not perform better than the original design, as expected given the point above.

W3. The revised text now states that guided prompting improves diversity in 11 of the 12 dataset × metric combinations (Sec. 4.7).

W4. Constructing imbalanced variants of standard benchmarks is common practice in imbalance-aware active learning (e.g., GALAXY [1], DIRECT [2], AnchorAL [3]), because it isolates the effect of the imbalance level from dataset size and domain (App. B); we follow it so that results are comparable across MeanIR levels and datasets. Naturally long-tailed benchmarks with a very large label space, in which many classes have only a handful of examples, form an extreme-classification regime that raises a different challenge and is stated as out of scope and future work in the conclusion. Computational cost is reported in Table 9: the query time of BIRDS is comparable to BEMPS, and IGR adds 2–4 minutes per round on a single RTX 3090. These 2–4 minutes include loading and unloading the LLMs (bfloat16, no quantization, one model resident at a time; App. C.7), so the overhead is modest.

[1] Zhang, J., Katz-Samuels, J., & Nowak, R. Galaxy: Graph-based active learning at the extreme. ICLR 2022.

[2] Nuggehalli, S., Zhang, J., Jain, L.K. & Nowak, R.D. Improved Algorithm for Deep Active Learning under Imbalance via Optimal Separation. ICML 2025.

[3] Lesci, P., & Vlachos, A. AnchorAL: Computationally efficient active learning for large and imbalanced datasets. NAACL 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.