acceptodds
Under review as a conference paper at ICLR 2027

Beyond Local Positives: LLM-Verified Diverse Positive Mining for Generalized Category Discovery

Abstract

Generalized Category Discovery (GCD) aims to recognize known categories and discover novel ones from unlabeled data given limited labeled examples of known categories. A central difficulty is constructing effective training signals for unlabeled instances, especially for novel categories. Recent LLM-assisted methods primarily improve the semantic reliability of these signals, but under-emphasize positive diversity: the coverage of same-class pairs that are less similar under the current representation. An oracle study with ground-truth filtering confirms this gap: even when all positives are same-class pairs, restricting them to highly similar candidates still limits discovery. However, expanding the candidate pool also introduces false-positive relations when ground-truth labels are unavailable. To address this trade-off, we propose DiPo (Diverse Positive mining), a budgeted two-path framework for neighborhood contrastive learning. For a fraction of anchors, DiPo expands the candidate pool and uses LLM same-or-different judgments to retain semantically valid, less-similar positives; for the remaining anchors, it uses the nearest cluster-consistent mutual neighbor as a reliable local positive. Extensive experiments show that DiPo attains the best average performance among strong LLM-assisted baselines and is particularly effective on novel-class discovery. Code is available at https://anonymous.4open.science/r/DiPo-5244DiPo.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Generalized Category Discovery, Contrastive Learning, LLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_64PJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the relationship between neighborhood size, neighborhood purity, and accuracy for neighborhood contrastive learning (NCL) as part of GCD training. Oracle experiments show that a larger neighborhood size k is beneficial when purity is high, however in practice purity decreases with increasing k. To solve this trade-off, DiPo is proposed, a GCD method that uses an LLM-assisted positive mining scheme to achieve high purity even for large k. The mined neighbor relations are then used for NCL training. To limit the LLM usage, only a fraction of samples have their neighbor relations verified, the other samples use the nearest mutual neighbor. Experiments show DiPo outperforming previous works on most standard text GCD benchmarks. Further analysis shows that LLM verification is key towards extracting accuracy gains from larger neighborhood sizes, with LLM-free variants not benefiting from increasing k.

Critical strengths
  • The core motivation of improving the diversity of the positive NCL pairs is sound and the oracle experiments clearly prove that positive diversity can be an important factor in improving GCD performance. This observation then underpins the proposed DiPo method, showing that it is sound and well-motivated.
  • The experiments are extensive, with detailed results for various ablations. In addition, the benefit of the LLM supervision is shown clearly and the synergy between neighborhood size and LLM-based positive mining is demonstrated. These results show that key design choices are justified.
Critical weaknesses
  • As shown in Table 18 of the paper, the choice of LLM has significant impact on the final results, however DiPo uses a different LLM compared to other methods such as DeLFGCD. As a result, the comparison to other methods is not fair, especially since the Qwen3.5-Flash model used for DiPo is generally considered superior to GPT-4o-mini used by DeLFGCD. Besides the choice of LLM, another important factor is how often the LLM is queried, Figure 7 shows that more LLM supervision further improves results compared to the default settings. The experimental setup does not consider this dimension, there is no comparison of query counts or token usage across methods, and no attempt to compare at matched LLM usage to isolate the impact of method design vs differences in LLM usage. Given that the gap between the two methods is rather small, the claim that DiPo outperforms previous methods can currently not be verified from the supplied experiments.
  • Beyond the positive construction, DiPo does not materially improve on previous work, the paper's novelty lies solely in the LLM-based mining of distant positives.
Questions for authors
  • How does DiPo perform when using less specific LLM prompts? The prompt template in Appendix F.6 includes dataset-specific examples, how does omitting them affect performance?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 64PJ

We sincerely thank the reviewer for the thorough evaluation and constructive feedback, which have helped us improve the clarity and rigor of our work. We address each comment point by point below.

W1: Fairness of the LLM and annotation budget.

Thank you for raising these concerns. We reran DeLFGCD using the default method settings reported in its paper and Qwen3.5-Flash, the same LLM used by DiPo. We adjusted DiPo’s annotation settings so that its total token usage stayed within that of the corresponding DeLFGCD run. The table reports token usage, estimated API costs, and clustering performance for both methods on BANKING, CLINC, and StackOverflow.

With the same LLM, DiPo achieves higher ACC at lower estimated API cost on all three datasets. The ACC gains are 2.86, 2.40, and 1.20 percentage points on BANKING, CLINC, and StackOverflow, respectively. DiPo uses fewer input and output tokens in every comparison. ARI and NMI also improve on BANKING and CLINC and are slightly lower on StackOverflow.

DatasetMethodInput tokensOutput tokensTotal tokensAPI cost (USD)ACC (%)ARI (%)NMI (%)
BANKINGDeLFGCD4,459,58766,0974,525,6840.1422474.5163.8784.83
BANKINGDiPo3,401,12850,8873,452,0150.1086177.3766.1985.29
CLINCDeLFGCD6,739,23672,0096,811,2450.2072090.2785.5295.70
CLINCDiPo1,554,56861,6371,616,2050.0603092.6787.4496.01
StackOverflowDeLFGCD1,752,65255,9611,808,6130.0642383.8070.3780.36
StackOverflowDiPo1,413,92348,5911,462,5140.0527785.0069.9980.35

Note. Both methods follow the same evaluation protocol at KCR = 25%. We estimate API cost from the recorded token counts using Qwen3.5-Flash rates of CNY 0.20 per million input tokens and CNY 2.00 per million output tokens, then convert to USD at a fixed reporting rate of CNY 7.20 per USD.

W2: Novelty beyond positive construction.

We appreciate the reviewer’s comment. Our central finding is that semantic reliability alone is insufficient for effective positive supervision in GCD; positive diversity also matters. We systematically investigate and substantiate this finding through controlled oracle experiments in Section 2. Guided by this finding, we propose DiPo to jointly address reliability and diversity under the limited labeled supervision available in GCD. DiPo achieves strong performance across three benchmarks, while the ablation studies further support this design. Thus, our contribution goes beyond simply using LLMs to mine distant positives: it is grounded in an empirical finding and translates this insight into an effective approach to addressing the reliability–diversity trade-off in positive supervision for GCD.

Q1: Prompts without domain examples.

Thank you for this suggestion. We evaluated Task-defined prompts that omit domain examples and retain a short category criterion. For BANKING and CLINC, the criterion begins with “Match the specific user goal or issue.” For StackOverflow, it begins with “Match the primary technology, platform, or product ecosystem the question is about.”

DatasetPromptACC (%)ARI (%)NMI (%)
BANKINGOriginal79.3869.5387.29
BANKINGTask-defined77.8668.5486.67
CLINCOriginal93.6989.5796.80
CLINCTask-defined92.8088.2196.31
StackOverflowOriginal87.6075.5882.30
StackOverflowTask-defined86.2073.0480.99

Note. Both prompt settings use KCR = 25%.

The shorter prompts retain most of the clustering performance, with ACC decreases of 1.52, 0.89, and 1.40 percentage points on BANKING, CLINC, and StackOverflow, respectively. The appendix subsection “Sensitivity to Prompt Specificity” provides the full Task-defined prompt templates, clustering results, and annotation metrics.

Thank you for providing these additional experiments. If time permits, could you please also run the task-defined prompt under the same cost-constraints as in the table you provided for W1?

AuthorsOpenReview ↗

Response to Reviewer 64PJ

We thank the reviewer for this helpful comment. We have now added the token usage, estimated API cost, and performance of the Task_defined prompt. Task_defined showed a slight performance decrease relative to Original under the default parameters but also consumed fewer tokens. To make the comparison fair under a comparable budget, we accordingly adjusted the annotation parameters for Task_defined. The table reports the results under this budget-matched setting. On BANKING, Task_defined achieves comparable performance at a notably lower cost, with only a marginal ACC trade-off while ARI and NMI are slightly higher. On CLINC and StackOverflow, the modest increase in cost is accompanied by small but consistent improvements across ACC, ARI, and NMI.

We thank the reviewer again for this constructive comment. It inspired us to consider streamlining the prompt so that more anchors can benefit from LLM-provided supervision, which may further improve our method. We believe this is a meaningful additional experiment.

DatasetMethodInput tokensOutput tokensTotal tokensAPI cost (USD)ACC (%)ARI (%)NMI (%)
BANKINGDeLFGCD4,459,58766,0974,525,6840.1422474.5163.8784.83
BANKINGDiPo(Original)3,401,12850,8873,452,0150.1086177.3766.1985.29
BANKINGDiPo(Task_defined)2,053,57872,2922,125,8700.0771276.9566.7285.32
CLINCDeLFGCD6,739,23672,0096,811,2450.2072090.2785.5295.70
CLINCDiPo(Original)1,554,56861,6371,616,2050.0603092.6787.4496.01
CLINCDiPo(Task_defined)1,947,48576,4352,023,9200.0753392.7687.9396.4
StackOverflowDeLFGCD1,752,65255,9611,808,6130.0642383.8070.3780.36
StackOverflowDiPo(Original)1,413,92348,5911,462,5140.0527785.0069.9980.35
StackOverflowDiPo(Task_defined)1,688,25075,3571,763,6070.0678386.272.2481.26
Reviewer_LSVYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the mechanism for selecting positive samples in GCD for text, based on neighborhood contrastive learning. The authors propose Dipo, which leverages a LLM to verify candidate samples filtered from a broader neighborhood. Experiments conducted on three benchmarks demonstrate the effectiveness of this approach.

Critical strengths
  1. Experiments show that each anchor contributes only one positive sample, and all positive samples belong to the same class according to the ground-truth labels. This setup effectively distinguishes between the diversity and the purity of the positive samples.

  2. The method is simple and cost-effective. The ablation studies are thorough.

Critical weaknesses
  1. The prompts used in Dipo are specifically designed for each individual benchmark. It is therefore difficult to determine whether performance improvements are from the diversity of positive samples or from the introduction of benchmark-specific LLM supervision.

  2. On the StackOverflow dataset (with KCR=10%), the reported accuracy is 74 for known classes and 76.56 for new classes. But the overall accuracy for the same experimental run is 79.1. The authors should verify these results and provide a clearer explanation of the experimental setup.

Questions for authors
  1. According to the computation method in Appendix A.2, the overall ACC should lie between the known-class and novel-class ACC, but some results in Tables 8 and 9 do not. Could the authors explain this?

  2. How will the results change with a same prompt shared across all benchmarks?

1 comment
AuthorsOpenReview ↗

Response to Reviewer LSVY

We sincerely thank the reviewer for the thorough evaluation and constructive feedback, which have helped us improve the clarity and rigor of our work. We address each comment point by point below.

W1 & Q2: Prompt specificity and a common prompt across datasets.

Thank you for this suggestion. We evaluated a Generic prompt with the following shared instructions across all three datasets:

For each numbered neighbor, decide whether it belongs to the same category as the anchor. Use a consistent level of detail across the comparisons. Different wording or incidental details can still belong to the same category. A broad connection alone is insufficient.

Return only a compact JSON array of all matching 1-based indices, or [] if none match.

DatasetPromptACC (%)ARI (%)NMI (%)
BANKINGOriginal79.3869.5387.29
BANKINGGeneric78.2168.2886.66
CLINCOriginal93.6989.5796.80
CLINCGeneric92.2286.8395.96
StackOverflowOriginal87.6075.5882.30
StackOverflowGeneric84.9069.8280.10

Note. Both prompt settings use KCR = 25%.

The generic prompt yields lower scores on all three datasets, with the largest ACC decrease on StackOverflow (2.70 percentage points). Its same-category recall on StackOverflow is 12.16%, suggesting that missed positive pairs may contribute to the larger performance drop. We provide the full annotation metrics in the appendix subsection “Sensitivity to Prompt Specificity”.

Separately, the controlled oracle experiments in Section 2 provide evidence for the benefit of positive diversity independently of LLM prompting. By using ground-truth labels and fixing one neighborhood positive per anchor, these experiments show that expanding the candidate neighborhood improves discovery performance even without LLM supervision.

W2 & Q1: Consistency of overall, known-class, and novel-class accuracy (Tables 8 and 9).

Thank you for pointing out this inconsistency. We identified an error in the computation of known- and novel-class ACC and corrected the affected results by re-evaluating the saved checkpoints from the original training runs, without retraining any models. We apologize for this error.

For StackOverflow at KCR = 10%, the corrected known-class, novel-class, and overall ACC are 73.00%, 79.78%, and 79.10%, respectively. These values satisfy the test-instance-weighted average relation under the global Hungarian mapping, up to rounding. We have clarified in Appendix A.2 that subset accuracies use the same Hungarian mapping obtained from the full test set.

We have corrected Tables 8 and 9 and updated the affected figures and discussion accordingly. DiPo continues to outperform NCL across all settings in Table 9. We have also revised the BANKING comparison at KCR = 25% to reflect that DiPo outperforms DeLFGCD in novel-class ACC and H-score, but not in known-class ACC. We thank the reviewer for helping us identify and correct this issue.

Reviewer_gaYbOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the Generalized Category Discovery (GCD) task in the NLP setting. This paper first identifies that recent LLM-assisted methods mainly improve the reliability of training signals for unlabeled data but ignore the diversity of positive samples for contrastive learning. To mitigate this issue, this paper introduces DiPo. By expanding the candidate pool and uses LLM same-or-different judgments to retain semantically valid, less-similar positives. Extensive experiments show that DiPo achieves SOTA performance on several text benchmarks.

Critical strengths
  1. The reliability-vs-diversity problem is well stated and validated by extensive experiments, making this paper well motivated.
  2. The paper is well written and easy to follow.
  3. Extensive experiments on several benchmarks validate the effectiveness of the proposed method.
  4. Ablation studies and other experiments verify the effectiveness of the components of the method.
Critical weaknesses
  1. Despite the improved performance, the main framework (neighborhood contrastive learning with LLM feedback) is from the previous study Loop, making the contribution of this paper incremental and limited.
  2. The paper did not report a comparison with other LLM-assisted methods with the same API budget, making the experimental comparison unfair.
  3. Per-dataset hyperparameter tuning raises overfitting concerns; Top-K is 100/120/500, and rho is 0.2/0.1/0.1. How to select these hyperparameters is not well stated.
  4. Results in Table 10 show that uniform sampling is statistically indistinguishable from other rules, so the effectiveness of the random anchor selection is not justified.
  5. Results in Table 1 (e.g., BANKING 79.02) and Table 3 (79.38) disagree slightly; the authors should explain it.
Questions for authors
  1. The paper addresses the GCD task under the NLP setting; however, most of the references are from the CV setting, and many GCD papers from the NLP area are not included. It would be better to include these NLP papers.
  2. Why not perform experiments under KCR = 75% setting and report the corresponding results?
4 comments
AuthorsOpenReview ↗

Response to Reviewer gaYb (Part 1/2)

We sincerely thank the reviewer for the thorough evaluation and constructive feedback, which have helped us improve the clarity and rigor of our work. We address each comment point by point below.

W1: Contribution beyond Loop.

We appreciate the reviewer's recognition of our motivation and controlled experiments. We acknowledge that DiPo builds on the LLM-assisted neighborhood contrastive learning framework used by Loop. However, while Loop uses LLM feedback to refine positives for locally inconsistent anchors, DiPo explicitly targets positive diversity by identifying reliable, less-local same-class positives through expanded candidate exploration and semantic verification. This distinction is motivated by the controlled oracle experiments in Section 2, which show that even with ground-truth-consistent positives and exactly one positive per anchor, restricting candidate selection to a narrow neighborhood can limit discovery performance. Table 2 further supports this design: under LLM verification, expanding the candidate pool improves ACC by 5.71, 6.89, and 3.80 percentage points on BANKING, CLINC, and StackOverflow, respectively.

Thus, our contribution is to identify the insight that semantic reliability alone is insufficient for effective positive supervision in GCD and that positive diversity also matters, and to translate this insight into an effective method that addresses the reliability–diversity trade-off in positive supervision for GCD.

W2: Fair comparison under annotation budgets.

Thank you for raising this concern about API budgets. We assess performance relative to API expenditure, allowing methods to retain their respective LLM configurations. We adjusted DiPo's annotation settings using each baseline's reported token usage or API cost as a reference and report the resulting token consumption, estimated costs, and clustering performance below.

Comparison with DeLFGCD. We used the token counts reported for DeLFGCD's default configuration to guide DiPo's annotation settings. DiPo achieves higher ACC on all three datasets while using fewer tokens, although ARI and NMI show mixed results. DiPo's estimated API costs are also lower, partly due to differences in the per-token pricing of Qwen3.5-Flash and GPT-4o-mini.

Comparison with LOOP. On CLINC and StackOverflow, DiPo improves all three clustering metrics within the same broad API cost range as LOOP. On BANKING, LOOP achieves higher performance at lower cost in the low-budget setting. At the higher budget, DiPo outperforms LOOP on all three metrics, achieving 1.59 percentage points higher ACC at approximately 13.5% lower estimated cost.

Overall, these comparisons provide a more transparent assessment of DiPo's performance and annotation costs across different budget scales. LOOP offers a better cost–performance trade-off in the lowest-budget BANKING setting, and DiPo achieves competitive or better performance in the other evaluated configurations.

DatasetMethodInput tokensOutput tokensAPI cost (USD)ACC (%)ARI (%)NMI (%)
BANKINGDeLFGCD3,631,78059,2400.5803176.9866.0085.62
BANKINGDiPo3,401,12850,8870.1086177.3766.1985.29
CLINCDeLFGCD5,816,21065,8500.9119491.5187.0796.27
CLINCDiPo1,554,56861,6370.0603092.6787.4496.01
StackOverflowDeLFGCD1,503,42052,7000.2571384.1071.0180.90
StackOverflowDiPo1,413,92348,5910.0527785.0069.9980.35

Note. DeLFGCD’s clustering results and API usage are taken from its original paper (GPT-4o-mini). The clustering results for both methods follow the same evaluation protocol at KCR = 25%. Costs use the respective model rates.

DatasetMethodInput tokensOutput tokensAPI cost (USD)ACC (%)ARI (%)NMI (%)
BANKING (low)LOOP236,0726,7080.0084270.4960.1083.68
BANKING (low)DiPo468,6423,7530.0140668.1857.0581.58
BANKING (higher)LOOP1,072,94929,6900.0380571.4360.0483.17
BANKING (higher)DiPo889,21529,6390.0329373.0261.7583.33
CLINCLOOP167,2985,9260.0062987.6981.7294.64
CLINCDiPo253,1036,3960.0088191.0284.7395.35
StackOverflowLOOP259,2318,1990.0094877.561.9276.08
StackOverflowDiPo304,0196,8990.0103679.9068.0279.69

Note. The original LOOP paper uses GPT-3.5 Turbo and does not report token usage. The original endpoint was unavailable for our reruns, so we measured LOOP’s token usage at KCR = 25% with Qwen3.5-Flash, the same model used by DiPo. We estimate Qwen3.5-Flash costs in both tables at CNY 0.20 per million input tokens and CNY 2.00 per million output tokens, using a fixed reporting rate of CNY 7.20 per USD.

AuthorsOpenReview ↗

Response to Reviewer gaYb (Part 2/2)

W3: Hyperparameter selection.

Thank you for highlighting the need to clarify our hyperparameter choices. The ground-truth oracle experiments in Section 2 show that expanding the neighborhood provides access to more diverse same-class positives and improves discovery performance, with gains gradually leveling off. Motivated by this observation, we use the average number of training instances per category, N/CN/C, as a reference for selecting the candidate neighborhood size, where NN is the number of training instances and CC is the total number of categories. For datasets with much larger classes, a smaller candidate pool may already provide sufficiently diverse positives. We have clarified this rationale in Section 4, alongside Fig. 7.

BANKING and CLINC use K=100K=100 and K=120K=120, respectively, close to their average class sizes of 117 and 120. StackOverflow uses K=500K=500, below its average class size of approximately 900. These choices are fixed across KCR settings and random seeds.

The revised Fig. 7(a–b) examines sensitivity to KK on BANKING and StackOverflow at KCR = 25%. On BANKING, performance improves as KK approaches the average class size and remains relatively stable around this scale. On StackOverflow, expanding the neighborhood beyond K=500K=500 does not yield further gains. These observations are consistent with our selection rationale. For clarity, the corresponding numerical results are also reported in Tables below.

For ρ\rho, we balance clustering performance and annotation cost. Fig. 7(c) shows diminishing gains beyond ρ=0.2\rho=0.2 on BANKING, supporting this choice. We use ρ=0.1\rho=0.1 on CLINC and StackOverflow to limit annotation overhead. These ratios are also fixed across KCR settings and random seeds.

  • BANKING, KCR = 25%
KACCARINMI
5073.6763.3784.90
8077.8967.4286.69
10079.3869.5387.29
12078.3869.2987.23
15078.4768.886.81
  • Stackoverflow, KCR = 25%
KACCARINMI
30083.8069.6280.65
50087.6075.5882.30
70084.969.7180.45
90080.467.8479.17

W4: Random anchor selection.

Thank you for raising this point. Our results suggest that a simple anchor selection strategy is sufficient to achieve competitive performance in DiPo. Uniform random sampling is straightforward to implement and avoids the additional computation and ranking required by entropy-based selection. As shown in Table 10, it performs comparably to both entropy-based alternatives on BANKING and CLINC under the same LLM query budget.

W5: Different results in Tables 1 and 3.

Thank you for pointing out this discrepancy. The difference arises from the number of random seeds used for reporting. Table 1 reports the mean over three seeds (79.02 on BANKING, as detailed in Table 15), whereas Table 3 reports a single-seed result (79.38) for the same default DiPo configuration. For computational cost, we use single-seed runs for ablation studies and report three-seed averages for the main comparisons. We have clarified this distinction in the caption of Table 3.

Q1: Coverage of NLP category discovery literature.

Thank you for this suggestion. We have expanded the Related Work discussion of text GCD and new intent discovery, adding DWGF (2023), TAN (2024), KTN (2024), and LR2GS (2026). The revised paragraph covers knowledge transfer, neighborhood-based positive construction, and local supervision for learning global intent structure.

Q2: Results at KCR = 75%.

Thank you for this suggestion. Our main comparison followed the KCR settings used by DeLFGCD. We have now added KCR = 75% results on all three datasets to Tables 8, 9, and 16. We also reran LOOP to compare clustering metrics with DiPo under the same evaluation protocol.

DatasetMethodOverall ACC (%)ARI (%)NMI (%)Known ACC (%)Novel ACC (%)H-score (%)
BANKINGLOOP80.4970.0987.3183.6270.9276.75
BANKINGDiPo83.5474.2588.7387.3371.9778.91
CLINCLOOP94.6291.3097.3896.4389.3092.73
CLINCDiPo97.0794.0697.9796.9097.5497.22
StackOverflowLOOP87.2075.1281.8586.4089.6087.97
StackOverflowDiPo88.8078.0683.3390.4084.0087.08

DiPo improves all three overall clustering metrics on each dataset. It also improves known- and novel-class ACC and H-score on BANKING and CLINC; on StackOverflow, LOOP retains higher novel-class ACC and H-score.

Thanks for your detailed responses. I decided to raise my score.

Thank you for your positive feedback and for raising the score. We are glad that our response addressed your concerns. Thank you again for your valuable comments and constructive suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.