acceptodds
Withdrawn from ICLR 2027

DiHypeSC: Data-Informed Hyperparameter Adaptation for Self-Expressive Subspace Clustering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Many subspace clustering methods require careful balancing of self-expressive reconstruction and structural regularization. Improper weighting can produce either dense cross-subspace connections or overly sparse and disconnected representations, both of which degrade clustering performance. However, the corresponding trade-off parameter is often selected through costly searches using labeled validation data, and its optimal value depends strongly on the dataset, noise level, representation, and regularizer. Label-free parameter selection remains challenging because deep clustering is stochastic and non-convex, while internal validation measures may not align with the ground-truth partition. We propose DiHypeSC, a data-informed strategy derived from subspace-preserving conditions that relate regularization strength to cross-subspace correlations and reconstruction residuals. During training, DiHypeSC estimates a proxy for the subspace dimension and uses it to adapt the regularization weight to each dataset. When integrated into two state-of-the-art deep subspace clustering methods, DiHypeSC provides a unified, dimension-aware weighting principle for both elastic-net and block-diagonal regularization. Even when using only a single hyperparameter configuration, DiHypeSC often achieves better results than conventional hyperparameter searches.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
unsupervised learning, self-expressive subspace clustering, hyperparameter optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GdLNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DiHypeSC, a data-informed method for adapting regularization weights \beta for both elastic-net and block-diagonal regularization in subspace clustering. It uses subspace-preserving conditions to guide β selection for elastic-net and block-diagonal regularization. Experiments suggest competitive clustering performance compared with random search and Bayesian optimization, with fewer training runs for hyperparameter tuning.

Critical strengths

The paper is well-written and technically sound. The motivation is clear. The proposed method has strong theoretical support.

Critical weaknesses

Theoretically:

  1. Line 212. The relative proof for Lemma 1 (and Lemma 5) is found in [2] page 3, not [1].
  2. Line 196 and 199. The estimator computes (\hat C) using the learned query/key embeddings in SENet, but not fit the standard formulation of C in the comming theories. It makes me confused.
  3. Theorem1. The restricted weighted ℓ1\ell_1 norm formulation introduces wijw_{ij} but not give any denotation, which will certainly affect the interpretability of β^i\hat \beta_i for Elastic-Net SC problem on line 266-267.
  4. Lemma 3. The notation of U: "a fixed Laplacian" makes me confused. Does U have any relation with L(C)\mathcal{L}(C)? Or just choose randomly.
  5. Lemma 3. As read the PRO-DSC, I think PRO-DSC mainly aims to balance the trade-off between SE loss and TCR loss(Expansion Loss), which belong to Deep Representation Learning topic. I recommend only talk BDR[3] purely in this theory.
  6. As mentioned in Theorem 1 and Lemma 2, author use weighted ℓ1\ell_1 norm to replace the Regularization items, which has similar structure studied in [4]. Therefore, I recommend to discuss the connection with [4] and gives more clear geometric meaning of wijw_{ij}.

Experiments:

  1. Algorithm 1,2. The c^\hat c is estimated only for βt\beta_t, by a matrix inverse computation. Is βt\beta_t computed every batch/epoch? I am concerned about the computation cost. Author should report the time comsumming proportion of the added estimation.
  2. Paper lacks the comparison with baseline setting of SENet and PRO-DSC performance reported in the raw paper, and also should report ACC/NMI for more fair comparison.
  3. The PRO-DSC method is sensitive to the hard dataset like CIFAR20, I recommend to add and report.
  4. Theorem 1, Lemma 3 lacks some sythetic experiments to support the conclusion.

I am willing to upgrade my score corresponding to the problem solved as mentioned above.

[1] Guangcan Liu, Zhouchen Lin, Shuicheng Yan, Ju Sun, Yong Yu, and Yi Ma. Robust recovery of subspace structures by low-rank representation. IEEE Trans. Pattern Anal. Mach. Intell., 35 (1):171–184, 2013. doi: 10.1109/TPAMI.2012.88. URL https://doi.org/10.1109/ TPAMI.2012.88.

[2] Mahdi Soltanolkotabi, Ehsan Elhamifar, and Emmanuel J. Cande`s. Robust subspace clustering. The Annals of Statistics, 42(2):669 – 699, 2014b. doi: 10.1214/13-AOS1199. URL https: //doi.org/10.1214/13-AOS1199.

[3] Canyi Lu, Jiashi Feng, Tao Mei, Zhouchen Lin and Shuicheng Yan. Subspace Clustering by Block Diagonal Representation. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2019.

[4] Nimita Shinde, Tianjiao Ding, Daniel P. Robinson, and René Vidal. Geometric Analysis of Nonlinear Manifold Clustering. In Annual Conference on Neural Information Processing Systems, 2024

Questions for authors

See above

Reviewer_FAoUOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes DiHypeSC, a method for adapting the regularization weight during deep self-expressive subspace clustering without performing a separate hyperparameter search. The method uses a pseudoinverse coefficient norm as a proxy for subspace dimension and applies this idea to SENet and PRO-DSC. The paper provides theoretical motivation and evaluates the method across several datasets, showing useful gains in some settings, especially for PRO-DSC, but the theoretical and reporting inconsistencies currently weaken the conclusions.

Critical strengths
  1. The paper addresses a practical problem in deep subspace clustering. Choosing the regularization weight can require expensive searches, and using representation geometry to adapt it during training is a useful idea.

  2. The connection between block-diagonal regularization and weighted sparsity is also interesting, and the method shows good improvements for PRO-DSC on some datasets while using a single run instead of repeated searches.

Critical weaknesses
  1. My main concern is the mathematical consistency of the paper. The proof of Theorem 1 appears to drop a factor of β\beta and later omits the ridge term in the KKT condition. The weighted regularization argument also appears to assume an additional residual bound rather than derive it.

  2. There are also inconsistencies in the practical method and results. Eq. 6 and Algorithm 3 do not appear to describe exactly the same rule, and some DiHypeSC results differ between Table 1 and Table 6 even though they seem to correspond to the same one-shot method. These issues make it difficult to know exactly what was evaluated.

  3. The paper does not compare against approaches that avoid regularization-weight tuning entirely, for example through closed-form clustering or projection-based solutions. Since the main motivation is to reduce the cost of tuning β\beta, this comparison seems important for understanding when adaptive tuning is actually preferable. Are the authors familiar with such approaches, and how would DiHypeSC compare against them?

Questions for authors
  1. Can the authors correct or clarify the proof of Theorem 1, especially the missing β\beta and ridge terms?

  2. Which exact adaptation rule was used for the reported PRO-DSC experiments, and how should the differences between Eq. 6 and Algorithm 3 be interpreted?

  3. Why do some DiHypeSC results differ between Table 1 and Table 6?

  4. Does the method remain reliable under noise, where the coefficient norm used to estimate the regularization weight can become much larger than predicted?

Reviewer_UqpSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors aim to address a practical problem: the performance of existing deep subspace clustering methods often depends heavily on the regularization coefficient. Experiments in the literature typically identify the optimal hyper-parameters by performing multiple training runs and calculating ARI or ACC using ground-truth labels. However, this approach conflicts with real-world unsupervised clustering scenarios, where ground-truth labels are usually unavailable. The authors propose DiHypeSC, which attempts to leverage the theoretical properties of subspace clustering to estimate the regularization coefficient from the geometric structure of the current data, and apply this method to two different deep subspace clustering algorithms.

Critical strengths
  1. Using ground-truth labels for hyperparameter tuning in deep unsupervised clustering does indeed present methodological contradictions in real-world applications.
  2. The overall paper is easy to follow and well organized.
  3. A comparison with internal unlabeled evaluation metrics is provided.
Critical weaknesses
  1. The authors need to explain the gap between the ideal linear subspace data discussed in Lemma 1 and the actual nonlinear mappings used in neural networks. For example, in SENet, the query and key are learned by two independent MLPs, and these two independent nonlinear mappings may not necessarily map them to the same linear subspace. Furthermore, the intermediate representations during the training of deep models may not preserve the “union-of-subspaces” structure of the original data, especially during the early stages of training.
  2. The theoretical conditions for the block-diagonal regularization defined by the authors seem to require knowledge of which samples belong to different subspaces, which cannot be directly obtained in unsupervised clustering.
  3. The branching conditions in the first two lines of Eq. (6) can hold simultaneously.
  4. The authors aim to eliminate the model’s reliance on hyper-parameter tuning. While the proposed method does reduce the search for regularization term coefficients, it introduces new hyper-parameters such as the constants a and b. Therefore, it appears to merely reformulate the hyper-parameters that require tuning.
  5. In the comparative experiments, the baselines also include some simple fixed rules, such as directly using SENet and the fixed β\beta values recommended in the PRO-DSC paper, or randomly selecting β\beta.
Questions for authors

Please refer to Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.