- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates the relationship between neighborhood size, neighborhood purity, and accuracy for neighborhood contrastive learning (NCL) as part of GCD training. Oracle experiments show that a larger neighborhood size k is beneficial when purity is high, however in practice purity decreases with increasing k. To solve this trade-off, DiPo is proposed, a GCD method that uses an LLM-assisted positive mining scheme to achieve high purity even for large k. The mined neighbor relations are then used for NCL training. To limit the LLM usage, only a fraction of samples have their neighbor relations verified, the other samples use the nearest mutual neighbor. Experiments show DiPo outperforming previous works on most standard text GCD benchmarks. Further analysis shows that LLM verification is key towards extracting accuracy gains from larger neighborhood sizes, with LLM-free variants not benefiting from increasing k.
Critical strengths
- The core motivation of improving the diversity of the positive NCL pairs is sound and the oracle experiments clearly prove that positive diversity can be an important factor in improving GCD performance. This observation then underpins the proposed DiPo method, showing that it is sound and well-motivated.
- The experiments are extensive, with detailed results for various ablations. In addition, the benefit of the LLM supervision is shown clearly and the synergy between neighborhood size and LLM-based positive mining is demonstrated. These results show that key design choices are justified.
Critical weaknesses
- As shown in Table 18 of the paper, the choice of LLM has significant impact on the final results, however DiPo uses a different LLM compared to other methods such as DeLFGCD. As a result, the comparison to other methods is not fair, especially since the Qwen3.5-Flash model used for DiPo is generally considered superior to GPT-4o-mini used by DeLFGCD. Besides the choice of LLM, another important factor is how often the LLM is queried, Figure 7 shows that more LLM supervision further improves results compared to the default settings. The experimental setup does not consider this dimension, there is no comparison of query counts or token usage across methods, and no attempt to compare at matched LLM usage to isolate the impact of method design vs differences in LLM usage. Given that the gap between the two methods is rather small, the claim that DiPo outperforms previous methods can currently not be verified from the supplied experiments.
- Beyond the positive construction, DiPo does not materially improve on previous work, the paper's novelty lies solely in the LLM-based mining of distant positives.
Questions for authors
- How does DiPo perform when using less specific LLM prompts? The prompt template in Appendix F.6 includes dataset-specific examples, how does omitting them affect performance?
5 comments
Thank you for providing these additional experiments. If time permits, could you please also run the task-defined prompt under the same cost-constraints as in the table you provided for W1?
Response to Reviewer 64PJ
We thank the reviewer for this helpful comment. We have now added the token usage, estimated API cost, and performance of the Task_defined prompt. Task_defined showed a slight performance decrease relative to Original under the default parameters but also consumed fewer tokens. To make the comparison fair under a comparable budget, we accordingly adjusted the annotation parameters for Task_defined. The table reports the results under this budget-matched setting. On BANKING, Task_defined achieves comparable performance at a notably lower cost, with only a marginal ACC trade-off while ARI and NMI are slightly higher. On CLINC and StackOverflow, the modest increase in cost is accompanied by small but consistent improvements across ACC, ARI, and NMI.
We thank the reviewer again for this constructive comment. It inspired us to consider streamlining the prompt so that more anchors can benefit from LLM-provided supervision, which may further improve our method. We believe this is a meaningful additional experiment.
| Dataset | Method | Input tokens | Output tokens | Total tokens | API cost (USD) | ACC (%) | ARI (%) | NMI (%) |
|---|---|---|---|---|---|---|---|---|
| BANKING | DeLFGCD | 4,459,587 | 66,097 | 4,525,684 | 0.14224 | 74.51 | 63.87 | 84.83 |
| BANKING | DiPo(Original) | 3,401,128 | 50,887 | 3,452,015 | 0.10861 | 77.37 | 66.19 | 85.29 |
| BANKING | DiPo(Task_defined) | 2,053,578 | 72,292 | 2,125,870 | 0.07712 | 76.95 | 66.72 | 85.32 |
| CLINC | DeLFGCD | 6,739,236 | 72,009 | 6,811,245 | 0.20720 | 90.27 | 85.52 | 95.70 |
| CLINC | DiPo(Original) | 1,554,568 | 61,637 | 1,616,205 | 0.06030 | 92.67 | 87.44 | 96.01 |
| CLINC | DiPo(Task_defined) | 1,947,485 | 76,435 | 2,023,920 | 0.07533 | 92.76 | 87.93 | 96.4 |
| StackOverflow | DeLFGCD | 1,752,652 | 55,961 | 1,808,613 | 0.06423 | 83.80 | 70.37 | 80.36 |
| StackOverflow | DiPo(Original) | 1,413,923 | 48,591 | 1,462,514 | 0.05277 | 85.00 | 69.99 | 80.35 |
| StackOverflow | DiPo(Task_defined) | 1,688,250 | 75,357 | 1,763,607 | 0.06783 | 86.2 | 72.24 | 81.26 |
Response to Reviewer 64PJ
We sincerely thank the reviewer for the thorough evaluation and constructive feedback, which have helped us improve the clarity and rigor of our work. We address each comment point by point below.
W1: Fairness of the LLM and annotation budget.
Thank you for raising these concerns. We reran DeLFGCD using the default method settings reported in its paper and Qwen3.5-Flash, the same LLM used by DiPo. We adjusted DiPo’s annotation settings so that its total token usage stayed within that of the corresponding DeLFGCD run. The table reports token usage, estimated API costs, and clustering performance for both methods on BANKING, CLINC, and StackOverflow.
With the same LLM, DiPo achieves higher ACC at lower estimated API cost on all three datasets. The ACC gains are 2.86, 2.40, and 1.20 percentage points on BANKING, CLINC, and StackOverflow, respectively. DiPo uses fewer input and output tokens in every comparison. ARI and NMI also improve on BANKING and CLINC and are slightly lower on StackOverflow.
Note. Both methods follow the same evaluation protocol at KCR = 25%. We estimate API cost from the recorded token counts using Qwen3.5-Flash rates of CNY 0.20 per million input tokens and CNY 2.00 per million output tokens, then convert to USD at a fixed reporting rate of CNY 7.20 per USD.
W2: Novelty beyond positive construction.
We appreciate the reviewer’s comment. Our central finding is that semantic reliability alone is insufficient for effective positive supervision in GCD; positive diversity also matters. We systematically investigate and substantiate this finding through controlled oracle experiments in Section 2. Guided by this finding, we propose DiPo to jointly address reliability and diversity under the limited labeled supervision available in GCD. DiPo achieves strong performance across three benchmarks, while the ablation studies further support this design. Thus, our contribution goes beyond simply using LLMs to mine distant positives: it is grounded in an empirical finding and translates this insight into an effective approach to addressing the reliability–diversity trade-off in positive supervision for GCD.
Q1: Prompts without domain examples.
Thank you for this suggestion. We evaluated Task-defined prompts that omit domain examples and retain a short category criterion. For BANKING and CLINC, the criterion begins with “Match the specific user goal or issue.” For StackOverflow, it begins with “Match the primary technology, platform, or product ecosystem the question is about.”
Note. Both prompt settings use KCR = 25%.
The shorter prompts retain most of the clustering performance, with ACC decreases of 1.52, 0.89, and 1.40 percentage points on BANKING, CLINC, and StackOverflow, respectively. The appendix subsection “Sensitivity to Prompt Specificity” provides the full Task-defined prompt templates, clustering results, and annotation metrics.