acceptodds
Under review as a conference paper at ICLR 2027

LSHC: LLM-Guided Semantic Hierarchical Clustering for Imbalanced and Multi-Granular Text Data

Abstract

Text clustering is a popular research topic that faces challenges such as multi-level granularity, cluster size imbalance, and local structure ambiguity in the embedding space. We propose LLM-Guided Semantic Hierarchical Clustering (LSHC), a bottom-up framework that combines embedding-based text clustering with semantic judgment from a large language model (LLM). LSHC constructs fine-grained initial clusters, which can preserve small semantic clusters. This makes the framework particularly suitable for imbalanced data. Then, an LLM is used to determine an appropriate optimization operation: MERGE, KEEP, or RECLUSTER. MERGE combines semantically similar micro-clusters, KEEP preserves distinct neighbor clusters, and RECLUSTER prevents error assignments from propagating to higher semantic levels by reorganizing micro-clusters with mixed semantics. Iteratively conducting these three operations constructs a semantic hierarchy and can be terminated according to the desired cluster granularity. Experiments on six benchmarks show that LSHC achieves notable improvements on several highly imbalanced datasets, particularly MASSIVE-Intent and MTOP-Intent.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Text Clustering, Unsupervised learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_S5M1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LSHC, an LLM-guided bottom-up clustering framework for imbalanced and multi-granularity text data. The method starts from an over-clustered fine-grained partition, selects candidate cluster pairs by centroid similarity, and asks an LLM to choose among MERGE, KEEP, and RECLUSTER using central and boundary representative examples. The authors report strong results with Gemma embeddings on MASSIVE-I, MASSIVE-D, and especially MTOP-I among six benchmarks. The ablation also shows that full LSHC generally outperforms a merge-only variant on MTOP-I and MASSIVE-I.

Critical strengths

S1: The method uses the LLM for local structural decisions over cluster pairs, while leaving large-scale sample assignment to numerical clustering in embedding space. The use of central and boundary examples provides complementary evidence and offers a plausible scalability advantage over per-instance LLM clustering.

S2: This operation provides a local correction mechanism beyond a merge-only bottom-up procedure. In Table 2, adding RECLUSTER yields substantial improvements on MTOP-I; with Gemma, ACC increases from 74.94 to 87.57. This provides empirical support for the module beyond its conceptual motivation.

Critical weaknesses

W1: Under Eq. (12), RECLUSTER permits samples to be reassigned across the two previous clusters. The updated partition therefore need not be a coarsening of the previous one: an old cluster may be split across multiple new clusters. This differs from the nested structure required by conventional agglomerative hierarchical clustering or a dendrogram. Since multi-granularity semantic hierarchy is a central contribution, the paper should clarify how the claimed hierarchy is represented, traced, and used after reclustering, or describe the output more precisely as a sequence of iterative multi-granularity partitions.

W2: Section 4 sets K_target to the ground-truth class count and initializes with twice that number of clusters. Although the paper notes that KEEP decisions and the maximum iteration limit can also terminate the process, the experiments do not show whether the semantic stopping criterion selects a meaningful final number of clusters when the true class count is unknown, or how sensitive results are to the initial cluster count. This limits the evidence for the claim of automatically supporting unknown semantic granularity

Questions for authors

Q1: When RECLUSTER repartitions samples from Ci and Cj into multiple new clusters, how are parent–child relations or another navigable hierarchy representation constructed across iterations? Is the stored H a strictly nested hierarchy, a general graph, or a set of independent intermediate partitions

Q2: Could the authors report results without using the ground-truth K_target? This would directly validate the semantic stopping rule and the multi-granularity claim.

Q3: The paper states that the design substantially reduces token use and inference time, but does not report LLM calls, token counts, or wall-clock runtime. Please report the number of LLM calls, total tokens, and runtime per dataset

Q4: Results on MTOP-I and MASSIVE-I are consistent with the motivation, but overall ACC/NMI does not directly establish that small classes are preserved or minority classes improve. Frequency-stratified results or an analysis tracing small clusters is absent

6 comments

We thank the reviewer for this important observation. We clarify that LSHC does not explicitly preserve or reconstruct strictly nested relationships. Instead, the current framework stores an ordered sequence of revised flat partitions.

Unlike conventional agglomerative clustering, where merging decisions are irreversible, LSHC allows RECLUSTER to reorganize samples within the union of two selected clusters. While MERGE preserves nested relationships, RECLUSTER may modify previous cluster boundaries, meaning that consecutive partitions do not necessarily satisfy strict hierarchical nesting. Therefore, (H) is neither a conventional dendrogram nor an explicitly constructed general graph. However, the sequence of the flat partitions may have some hierarchical structure, since the merge operations create higher-level partitions.

The bottom-up design refers to progressively aggregating fine-grained clusters while processing local assignment correction. This design prevents potentially incorrect assignments from being permanently propagated through subsequent merging operations. We have clarified that the current framework produces a sequence of semantically refined partitions at potentially different resolutions, rather than a strictly nested hierarchy.

We appreciate the reviewer's suggestion. To directly investigate the dependence on K_target, we conduct an additional experiment on the complete MTOP-Intent dataset, containing 15,638 samples and 102 annotated categories, without enforcing the ground-truth target cluster number.

Starting from 200 initial micro-clusters, LSHC automatically terminates after 77 iterations when the five most similar candidate cluster pairs receive KEEP decisions. The final result contains 124 clusters, rather than the 102 annotated categories, and achieves 79.45 ACC and 90.52 NMI.

These results demonstrate that the proposed semantic stopping criterion can terminate the clustering process and produce a high-quality partition without requiring the ground-truth final cluster count. The difference between the discovered and annotated cluster counts further confirms that termination is not constrained to reproduce a predefined number of categories.

Regarding initialization, setting (K_0=2*K_target) is an experimental over-clustering configuration rather than a requirement of the proposed operation-selection mechanism. In a fully unsupervised setting, (K_0) can instead be specified independently or estimated from unlabeled embedding-space structure. In the current framework, semantic resolution is influenced primarily by the initial clustering granularity. A larger initial cluster number produces finer micro-clusters, allowing the LLM to examine subtle semantic distinctions during subsequent aggregation. Conversely, a smaller initial cluster count encourages broader initial groups and can support coarser semantic clustering.

Class frequency# Classes# SamplesACC (%)NMI (%)
Smallest 25%2618971.9685.91
25–50%2663083.6593.83
50–75%251,47781.4591.14
Largest 25%2513,34279.7989.85
Overall10215,63879.4590.52

We thank the reviewer for highlighting the importance of computational efficiency. To provide quantitative evidence, we add a comparison of LLM calls on the Massive dataset (Table 7).

Specifically, ClusterLLM requires 1,618 LLM calls, LLMEdge requires 892, and k-LLMmeans requires 90, whereas LSHC requires only 69 LLM calls. This corresponds to approximately 95.7%, 92.3%, and 23.3% fewer LLM calls, respectively.

This efficiency in call frequency follows from our lightweight semantic decision mechanism. Rather than asking the LLM to perform sample-level assignments, LSHC uses the LLM only to select MERGE, KEEP, or RECLUSTER based on representative cluster samples. Numerical clustering algorithms subsequently perform local sample reassignment without additional LLM inference.

ClusterLLMLLMEdgek-LLMmeansLSHC (Ours)
LLM Calls16188929069
Reviewer_HyyqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes LSHC, an LLM-guided hierarchical text clustering framework designed to address cluster size imbalance, assignment errors, and varying semantic granularity. The method initializes fine-grained clusters using K-means and iteratively selects neighboring cluster pairs based on centroid similarity. Representative samples from cluster centers and boundaries are provided to an LLM, which determines one of three operations: MERGE, KEEP, or RECLUSTER. The latter uses spectral clustering to reorganize potentially misassigned samples. Experiments on six text clustering benchmarks with three embedding models show promising improvements over traditional and LLM-guided clustering methods, particularly on highly imbalanced datasets such as MTOP-Intent and MASSIVE-Intent.

Critical strengths

1. Intuitive and practical LLM-guided clustering framework.

The proposed framework offers a reasonable approach to combining embedding-based clustering with LLM semantic judgments. In particular, selecting representative center and boundary samples to guide MERGE, KEEP, and RECLUSTER decisions is an intuitive and potentially useful design. It allows the framework to incorporate semantic reasoning while retaining the efficiency of traditional clustering algorithms.

2. Promising empirical results on imbalanced datasets.

The method achieves substantial improvements on challenging, highly imbalanced datasets, particularly MTOP-Intent and MASSIVE-Intent. The consistent gains across multiple embedding models on MTOP-Intent suggest that the approach has practical potential and addresses a meaningful limitation of conventional clustering methods.

Critical weaknesses

1. Insufficient validation of the core methodology.

The ablation study mainly evaluates the contribution of RECLUSTER, leaving other central components largely unexamined. In particular, the paper does not establish whether center-and-boundary sampling improves LLM decisions or whether the LLM can reliably identify when reclustering is appropriate. Although the overall results are promising, the limited analysis makes it difficult to understand which components drive the improvements and whether the proposed design is effective for its intended reasons.

2. Insufficient justification of key design choices.

The framework relies on several heuristics, including the 2× over-clustering ratio, the number of representative samples, the top-five KEEP stopping criterion, and eigengap-based reclustering. These choices receive little justification or sensitivity analysis. Given their potential influence on clustering outcomes, the lack of systematic investigation raises concerns about the robustness and generalizability of the method.

Questions for authors
  1. LLM decision effectiveness: Is there evidence that the LLM reliably distinguishes MERGE, KEEP, and RECLUSTER, particularly when identifying incorrect assignments? How can the reported improvements be attributed to the LLM's semantic judgments rather than the reclustering operation itself?

  2. Dependence on target cluster count: The experiments set the target number of clusters to the ground-truth number of classes and initialize with twice that number. How would these values be determined in a fully unsupervised setting, and to what extent does the method depend on knowing the true number of clusters?

3 comments

We thank the reviewer for raising this important concern. The primary role of the LLM in LSHC is to determine whether the operation of two candidate clusters is MERGE, KEEP, or RECLUSTER based on their core and boundary semantics. Unlike directly assigning samples, the LLM only selects the appropriate operation, while clustering algorithms perform sample reassignment. The case study also presents the effectiveness of each operation (KEEP, MERGE, and RECLUSTER).

To investigate whether the improvements originate from clustering algorithms, we extend our ablation study (Table 3) with a No LLM decision variant and two LSHC variants using k-means and spectral clustering, respectively. On MTOP-I with Gemma embeddings, the No LLM decision variant achieves 83.63 ACC and 91.43 NMI, whereas LSHC achieves 86.39/92.12 with k-means and 87.57/92.44 with spectral clustering. On MASSIVE-I, LSHC with k-means improves ACC from 72.63 to 77.37 and NMI from 82.36 to 83.31 compared with the No LLM decision variant. Therefore, the improvements are the same when replacing spectral clustering with k-means, demonstrating that the proposed refinement mechanism does not rely exclusively on spectral clustering. These results provide additional empirical support for the contribution of LLM-guided semantic decisions to overall clustering performance.

MethodEncoderMTOP-I ACCMTOP-I NMIMassive-I ACCMassive-I NMI
Merge onlyE551.8973.8554.2070.61
Merge onlyINSTRUCTOR48.2470.3655.7270.10
Merge onlyGemma74.9488.6770.9182.71
No LLM decisionE549.7572.7550.4468.57
No LLM decisionINSTRUCTOR53.6070.0457.4671.32
No LLM decisionGemma83.6391.4372.6382.36
LSHC (k-means)E560.6776.5456.6670.88
LSHC (k-means)INSTRUCTOR53.1969.8259.1871.83
LSHC (k-means)Gemma86.3992.1277.3783.31
LSHC (spectral clustering)E558.5075.4357.7071.41
LSHC (spectral clustering)INSTRUCTOR54.1069.9457.6371.05
LSHC (spectral clustering)Gemma87.5792.4474.6883.00

We appreciate the reviewer's suggestion. We clarify that the ground-truth number of classes is used as KtargetK_target in the main experiments to establish a known-cluster-count benchmark protocol. Ground-truth sample labels are not used during cluster-pair selection, LLM operation decisions, or local reclustering.

To directly investigate dependence on KtargetK_target, we conduct an additional experiment on the MTOP-Intent dataset, containing 15,638 samples and 102 annotated categories, without enforcing the ground-truth target count.

Starting from 200 initial micro-clusters, LSHC automatically terminates after 77 iterations when the top five candidate cluster pairs consistently receive KEEP decisions. The final partition contains 124 clusters, rather than the 102 ground-truth categories, and achieves 79.45 ACC and 90.52 NMI.

These results demonstrate that the semantic stopping rule can terminate the clustering process and produce a high-quality partition without requiring the ground-truth final cluster number. The final cluster count is determined by the semantic stopping mechanism rather than being forced to match the annotation taxonomy.

Regarding initialization, (K_0=2*K_target) is an experimental over-clustering configuration, not an intrinsic requirement of the operation-selection mechanism. In a fully unsupervised setting, (K_0) could instead be selected using unlabeled embedding-space structure or a predefined micro-cluster size budget.

Class frequency# Classes# SamplesACC (%)NMI (%)
Smallest 25%2618971.9685.91
25–50%2663083.6593.83
50–75%251,47781.4591.14
Largest 25%2513,34279.7989.85
Overall10215,63879.4590.52

I appreciate the authors' detailed responses and the additional experiments. The new ablations provide stronger evidence for the contribution of LLM-guided decisions, and the experiment without ground-truth K_target partially addresses my concerns about the unsupervised setting. I am therefore raising my rating from 2 to 3.

Reviewer_h5S4OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LSHC, an LLM-guided text clustering framework designed to address class imbalance, clustering error propagation, and multi-granularity semantic structures. The method first uses K-means to generate fine-grained initial clusters, then selects candidate cluster pairs based on centroid similarity. Representative central and boundary samples are provided to an LLM, which chooses between merging, keeping the clusters separate, or locally reclustering them; reclustering is implemented using spectral clustering. The authors evaluate the method across six text clustering settings and three embedding models, comparing it with traditional and LLM-assisted clustering methods using ACC and NMI. The results show improvements on MTOP-Intent and MASSIVE-Intent, but performance is inconsistent across the other datasets. An ablation study removing RECLUSTER assesses the contribution of local reclustering, while a case study illustrates the execution of the three operations.

Critical strengths

With the embedding model held fixed, the method achieves substantial improvements on MTOP-Intent, and the ablation study shows that adding RECLUSTER further improves performance. This provides empirical evidence for the effectiveness of local reclustering on this task and constitutes the paper’s most convincing positive result.

Critical weaknesses

(1) Limited methodological novelty. The method primarily combines K-means initialization, candidate selection based on centroid similarity, LLM-based semantic judgments, and local spectral clustering. The authors have not sufficiently established what substantive methodological advance the combination of MERGE, KEEP, and RECLUSTER provides over existing LLM-guided clustering and boundary-refinement approaches. The ablation study demonstrates that reclustering helps, but does not distinguish the benefits of local spectral clustering from the contribution of LLM-based operation selection. Consequently, the reported performance improvements are insufficient to establish the methodological contribution.

(2) The central claims regarding hierarchical structure and multi-granularity capabilities are insufficiently supported. RECLUSTER in Eq. (12) allows samples to be reassigned across previous cluster boundaries, so successive partitions are not necessarily nested. The algorithm does not explain how a consistent hierarchy is maintained or reconstructed. Furthermore, the prompt requires preserving fine-grained semantic distinctions but does not explain how the decision criteria adapt to the target granularity. Reporting separate intent-level and domain-level clustering results does not demonstrate that a structure constructed in a single run can support both levels.

Questions for authors

(1) What is the specific methodological innovation over existing LLM-guided clustering and boundary-refinement approaches? How does the current ablation distinguish the benefits of local spectral clustering itself from the contribution of LLM-based operation selection?

(2) RECLUSTER allows samples to be reassigned across previous cluster boundaries. How are nested relationships between levels preserved or reconstructed in this case? Does the algorithm output a hierarchy or a sequence of revised flat partitions?

(3) Given that the prompt explicitly requires preserving fine-grained distinctions such as ON/OFF, how are the decision criteria adjusted when a coarser semantic granularity is desired? Are the reported intent-level and domain-level results derived from a structure constructed in a single run?

9 comments

We thank the reviewer for this important question. We address your questions below, and the manuscript has been revised in blue.

The methodological innovation of LSHC lies in integrating LLM-guided semantic operation selection with iterative bottom-up clustering, rather than introducing a new numerical clustering algorithm. Unlike ClusterLLM, which uses LLM supervision for representation learning, or LLMEdgeRefine, which focuses on boundary-point refinement, LSHC employs an LLM to determine three cluster-level operations: MERGE, KEEP, and RECLUSTER. These decisions are based on representative samples from both cluster centers and boundaries, allowing the framework to distinguish semantically similar clusters, meaningfully distinct clusters, and potentially incorrect assignments.

To further isolate the contributions of LLM guidance and the local clustering algorithm, we extend Table 3 with two additional ablations: a No LLM decision variant and a variant replacing spectral clustering with k-means for RECLUSTER.

The results demonstrate that the benefits are not restricted to spectral clustering. On MTOP-I with Gemma embeddings, LSHC achieves 86.39 ACC using k-means and 87.57 using spectral clustering, compared with 74.94 for merge-only. More importantly, the No LLM decision variant achieves 83.63 ACC, below both LLM-guided variants. Similar observations hold on MASSIVE-I, where LSHC with k-means achieves 77.37 ACC, compared with 72.63 without LLM decisions and 70.91 for merge-only.

Overall, these ablation studies demonstrate that the effectiveness of LSHC is the combination of semantic operation selection and local assignment correction, rather than any individual component. The RECLUSTER operation is flexible to different clustering algorithms and provides decent performance with different embedding methods.

MethodEncoderMTOP-I ACCMTOP-I NMIMassive-I ACCMassive-I NMI
Merge onlyE551.8973.8554.2070.61
Merge onlyINSTRUCTOR48.2470.3655.7270.10
Merge onlyGemma74.9488.6770.9182.71
No LLM decisionE549.7572.7550.4468.57
No LLM decisionINSTRUCTOR53.6070.0457.4671.32
No LLM decisionGemma83.6391.4372.6382.36
LSHC (k-means)E560.6776.5456.6670.88
LSHC (k-means)INSTRUCTOR53.1969.8259.1871.83
LSHC (k-means)Gemma86.3992.1277.3783.31
LSHC (spectral clustering)E558.5075.4357.7071.41
LSHC (spectral clustering)INSTRUCTOR54.1069.9457.6371.05
LSHC (spectral clustering)Gemma87.5792.4474.6883.00

We appreciate the reviewer's observation and clarify that LSHC does not explicitly preserve or reconstruct strictly nested relationships.

Unlike conventional agglomerative clustering, LSHC permits local reassignment through RECLUSTER to correct potentially inaccurate cluster boundaries. Although MERGE combines two clusters into their union and naturally preserves a nested relationship, RECLUSTER repartitions samples within the union of the selected clusters. Consequently, the resulting clusters may cross previous cluster boundaries, and strict nesting between consecutive partitions is not guaranteed.

Specifically, Algorithm 1 indicates that HHstores the clustering partition after each iteration. Therefore, the output is an ordered sequence of revised flat partitions, rather than a strictly nested dendrogram or a general graph with explicit parent–child links. However, the sequence of the flat partitions may have some hierarchical structure, since the merge operations create higher-level partitions.

We emphasize that the bottom-up design describes the iterative refinement process, in which fine-grained initial clusters are progressively aggregated while local errors can be corrected before further merging. This differs from traditional irreversible hierarchical clustering. The advantage is greater flexibility in correcting assignments. The trade-off is that strict hierarchical nesting is not maintained. We will clarify this distinction in the method description and scope of the multi-granularity claim.

We thank the reviewer for this question. The fine-grained distinctions in our prompt, such as ON/OFF and SEND/RECEIVE, are intended to encourage the LLM to recognize subtle but meaningful semantic differences that embedding-based clustering may overlook. These examples are particularly important for intent discovery, where semantically related actions can represent different user intentions. They are illustrative criteria for KEEP decisions, rather than unconditional restrictions on merging such categories.

In the current framework, semantic resolution is influenced primarily by the initial clustering granularity. A larger initial cluster number produces finer micro-clusters, allowing the LLM to examine subtle semantic distinctions during subsequent aggregation. Conversely, a smaller initial cluster count encourages broader initial groups and can support coarser semantic clustering.

We clarify that the current implementation does not explicitly change the LLM decision criteria through a granularity-conditioned prompt. The wording “can require KEEP” is appropriate because it does not prohibit merging ON/OFF-related groups under all circumstances. Instead, different initial clustering resolutions provide different starting points for semantic refinement. In the current implementation, intent-level and domain-level clustering are conducted independently using their corresponding initialization settings. Therefore, the framework produces clustering results at different granularities in each individual run, rather than a strictly nested hierarchy reconstructed from a single execution.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Comment

We thank all reviewers for their careful review and useful suggestions. We have updated the paper to address all the questions in the reviews, with changes marked in blue. The main revisions are:

  1. Abstract, Sec. 1, Sec. 3.1, and Algorithm1: clarify the output clustering type. (h5S4, S5M1)
  2. Sec. 3.4: explain the prompt. (h5S4)
  3. Sec. 3.6 and Sec. 4: explain the initial number of micro-clusters. (Hyyq, S5M1)
  4. Sec. 4.2 and A.6: add the experiments to test Effectiveness under Cluster-Size Imbalance. (S5M1)
  5. Sec. 4.3, A.4, A.8, and Sec. 6: extend the ablation study for each component. (h5S4, Hyyq, S5M1)
  6. A.3: pure unsupervised clustering without target K. (Hyyq, S5M1)
  7. A.7: explore the LLM efficiency. (S5M1)
AuthorsOpenReview ↗

Follow-up on Rebuttal and Reviewer Discussion

Dear AC, SAC, and PC,

We sincerely thank Reviewer Hyyq for acknowledging that our rebuttal has addressed their concerns and for raising their rating.

We are still looking forward to responses from Reviewers S5M1 and h5S4. As Reviewers S5M1 and h5S4 raised several concerns similar to those of Reviewer Hyyq, we sincerely hope that the clarifications and additional evidence provided in our rebuttal have also helped address their concerns. We would greatly appreciate the opportunity for further discussion and hope that the remaining reviewers will reconsider their assessments

We appreciate your time and efforts in facilitating the review process.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.