acceptodds
Under review as a conference paper at ICLR 2027

Structure-Guided Transformer Clustering for High-Dimensional Data

Abstract

Clustering high-dimensional data is challenging when informative signals are distributed across dependent features and cluster structure is nonlinear. We introduce Structure-Guided Transformer Clustering (SGTC), which estimates a feature conditional-dependence graph, partitions related variables into groups, guides the splitting of Transformer heads, and learns a prototype-based latent partition. Across simulations, SGTC achieved higher agreement with ground-truth clusters than an unstructured Transformer, K-means, spectral clustering, and UMAP-based clustering, particularly as dimensionality increased. Its advantage was strongest when the sample size was sufficient to estimate useful dependence structure and diminished in sample-limited settings. SGTC showed robust performance under random edge deletions and additions to the feature-dependence graphs. We further applied SGTC to miRNA expression profiles from 418 patients with lung squamous cell carcinoma in The Cancer Genome Atlas. Among the evaluated methods, the clusters identified by SGTC showed the strongest overall survival separation (log-rank p=0.0066). The pathway enrichment analysis of the top driven miRNAs suggested more plausible pathways based on SGTC than the classic transformer.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
high-dimensional clustering, unsupervised learning, representation learning, structure-guided learning, feature dependence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zCZ5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a clustering method for high-dimensional data, specifically for cases where the number of features is larger than that of instances. The proposed method first identifies the dependence structure among features based on graph spectral analysis, and then utilizes the identified information for clustering instances. The performance of the proposed method was evaluated using synthetic datasets, demonstrating its robustness against feature graph perturbation. In addition, real biological datasets were analyzed using the proposed method as practical applications.

Critical strengths

The authors address an important problem setting in clustering where the number of features exceeds the number of instances. While the proposed method leveraging feature grouping yields promising results compared to existing baselines in numerical experiments, I have several concerns regarding the current manuscript.

Critical weaknesses
  1. The core contribution of the proposed method appears to be a structured Transformer leveraging feature groups. However, the effectiveness of this framework has not been sufficiently evaluated, either theoretically or empirically.

  2. Much of the necessary information is missing from the manuscript, such as variable definitions and the baseline methods. The unstructured Transformer should be explained in more detail in Section 4.2, specifically clarifying which part differs from the proposed method.

Questions for authors
  • Why is the permutation of observed variables assumed? It seems unlikely that such a scenario arises in practical applications. Could the authors provide practical examples where such a permutation occurs? In addition, Proposition 1 is rather trivial and may not warrant being designated as a standalone proposition.

  • Theorem 1 appears to cover the scenario where prototype labels are invariant. However, my understanding was that the underlying theory applies to a broader setting, unconstrained by specific feature groupings or transformer-based models. Could the authors clarify the generalizability of Theorem 1 regarding this aspect?

  • How did the proposed method adjust the number of principle components, described in line 140? It provides important information because the adjustment significantly affects to the clustering performance.

  • Could the authors clarify the rationale behind selecting the baseline/comparison methods? It would be valuable to include baselines that apply dimensionality reduction (e.g., PCA, spectral clustering, or UMAP) directly to unstructured Transformer inputs prior to K-means clustering. Such ablation baselines are essential to isolate and demonstrate the effectiveness of feature grouping.

Explanations of important variables are missing from the manuscript.

  • Line 110: Σ^LW\hat{\Sigma}_{LW} is covariance for which variables? (I guess it for features.)
  • Line 141 and 155: Variable pp is defined for different meanings.
  • Line 202: What is CDKC_{DK}?
  • Line 222: What is ρ\rho? What were replaced using this parameter? The data generation procedure is not understandable from this context.
  • Line 248: What is sparsity level ss?
  • Line 324: What are the horizontal and vertical axes?
Reviewer_8UJfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the high-dimensional clustering problem with dependent features and a nonlinear clustering structure. The authors present a four-step procedure:

  1. (their contribution) Estimate the feature covariance estimation matrix, then find the general inverse matrix as the estimated precision matrix. Use the precision matrix as a graph adjacency (weighted version and unweighted version) to cluster the features into H groups.
  2. (their contribution) For every data point, generate H heads. Each head h contains two parts: 4 summary statistics from features in group h, and a truncated PCA embedding of this data point. The combination requires a tuning parameter α\alpha.
  3. Use K-means to cluster the heads in Step 2, so that we have the pseudo-label.
  4. Train a transformer based on the pseudo-label in Step 3 and heads in Step 2.
Critical strengths

Combines the graph structure with transformers.

Critical weaknesses
  1. Insufficient explanation on this mechanism.
  2. Insufficient numerical comparisons.

I leave the details to questions, due to many questions I have.

Questions for authors
  1. The indices hh in pi(h)p_i^{(h)} (global PCA score blocks) and VhV_h (feature groups) have no natural correspondence. Why pairing pi(h)p_i^{(h)} with bˉih\bar b_{ih} in the same head?
  2. What is the role of head splitting?
  3. The graph summary is condensed into 4 summary statistics. Is that too simplified?
  4. In numerical comparisons, the authors only consider classical K-means, spectral clustering using the affinity matrix, and UMAP. Since m >> n, why not consider sparse clustering methods, at least sparse K-means and sparse spectral clustering? Also, why DEC is not included?
  5. The transformers uses the pseudo-label in Step 3 to train. Will it cause error propogation?
  6. Sensitivity to tuning parameters, such as the embedding dimension d_{PCA}, the feature groups H, and the tuning parameter α\alpha.
  7. In real data analysis, it seems SGTC-A doesn't perform well.
  8. Finally, a quick theoretical question: the inverse of Ledoit-Wolf shrinkage covariance estimate should be quite dense. Using the binary version will probably cause a complete connected network A. Does it happen in simulation and real data? How do you deal with it?

Writing problems:

  1. There are two ?? in the article, when referring to appendix. Please doule check.
  2. The presentation of Section 3.3-3.4 should be improved.
Reviewer_AnJ7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Structure-Guided Transformer Clustering (SGTC) for high-dimensional data. It estimates a feature-dependence graph, partitions features into balanced groups, and combines group-level statistics with global PCA scores as inputs to Transformer heads. Training uses a prototype-based objective with periodically refreshed K-means pseudo-labels. The paper provides conditional stability results and evaluates the method through synthetic experiments and a TCGA-LUSC miRNA application involving 418 patients. The reported results suggest substantial clustering improvements when sufficient samples are available.

Critical strengths
  1. A relevant and interpretable structural inductive bias. Organizing dependent features before representation learning is well motivated for high-dimensional molecular data. Estimating this structure directly from observations makes the approach applicable without requiring a predefined biological network.
  2. Promising reported performance across multiple experimental conditions. The synthetic evaluation uses 50 paired replicates and varies dimensionality, sample size, dependence, and graph perturbations. At n=500,m=2500, SGTC-Σ\Sigma achieves an ARI of 0.974 compared with 0.097 for the Transformer baseline. The explicit reporting of deterioration in small-sample settings helps establish the method’s practical limitations.
Critical weaknesses
  1. The stated binary graph construction appears inconsistent with its intended role. Section 3.2 defines an edge whenever the estimated precision entry is nonzero. Ledoit–Wolf shrinkage generally produces a dense precision matrix, so this construction would typically yield a complete binary graph. Such a graph cannot distinguish feature dependencies and leaves no absent edges for the reported edge-addition experiments. An unreported threshold or sparsification step could explain this discrepancy, but the current description does not establish what was evaluated.
  2. The comparisons do not isolate the contribution of graph guidance. According to Table 3, SGTC receives 30 PCA scores plus group statistics, whereas the Transformer baseline receives standardized original features. Their initialization, learning rates, head counts, and maximum training epochs also differ. Consequently, the reported gains cannot be attributed specifically to meaningful feature structure. Comparisons against matched preprocessing, random balanced groups, and simple clustering of the constructed representation are essential.
Questions for authors
  1. What exact thresholding or sparsification procedure was used to construct the graphs? Please report graph densities, the candidate set for selecting H, and explain how previously absent edges were selected for the edge-addition experiments.
  2. Does SGTC retain its advantage when the Transformer baseline receives the same PCA preprocessing and is evaluated with comparable training budgets and tuning?
  3. How do random balanced feature groups compare with estimated graph groups? Additionally, how well does K-means perform directly on the PCA-plus-summary representation before neural training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.