acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Residual Guidance for Multi-View Clustering

Abstract

Deep multi-view clustering (MVC) commonly relies on stacked view-specific encoders to learn increasingly abstract representations. However, deeper encoders do not necessarily produce better clustering features, because repeated nonlinear transformations can progressively weaken useful structures established at earlier stages. We propose Hierarchical Residual Guidance for Multi-View Clustering (HRG-MVC), which organizes deep multi-view representation learning around a hierarchical residual encoder. At each stage, the projected input is retained through an additive shortcut and the nonlinear branch learns only an incremental update, providing an explicit information path across encoder depth. To complement this view-specific residual hierarchy, a Cross-View Transformer estimates level-wise consensus from the representations of the same sample across views. The consensus is projected and selectively introduced into the subsequent residual stage through a feature-wise gate, allowing complementary cross-view information to refine deeper features without replacing their local basis. Reconstruction, inter-view feature contrast, and inter-level assignment contrast jointly optimize the hierarchy. Experiments on six benchmark datasets show competitive clustering performance, while depth and stress-test studies demonstrate the improved stability of residual encoders as the network becomes deeper. Additional ablations characterize the role of Transformer-based consensus and selective cross-view feedback.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
hierarchical residual guidance, transformer, multi-view clustering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JAJbOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the representation degradation problem in deep multi-view clustering, where increasing the encoder depth does not necessarily lead to better clustering performance. To address this issue, the authors develop HRG-MVC, which maintains information propagation through hierarchical residual connections while leveraging cross-view consensus to refine the learned representations. The consensus is estimated at different levels using a Transformer and adaptively incorporated into subsequent layers through a gating mechanism. The ablation studies and depth comparisons demonstrate the effectiveness of residual learning and the complementary role of cross-view guidance.

Critical strengths

1.The combination of hierarchical residual learning and cross-view guidance is interesting and well designed. In particular, incorporating consensus information into subsequent residual stages provides an effective way to enhance information interaction across different views. 2.The method achieves strong clustering performance on several benchmark datasets. The comparison between plain and residual encoders, together with the guidance ablation, provides good empirical support for the proposed framework.

Critical weaknesses

1.The effectiveness of cross-view guidance varies across datasets. Some additional explanation of this behavior would help readers better understand the role of the proposed gating mechanism. 2.While the theoretical complexity is discussed, the practical computational overhead introduced by the Cross-View Transformer and gating modules is not explicitly quantified.

Questions for authors

1.According to Table 4, gated guidance substantially improves performance on Caltech-5V but offers limited benefits on Handwritten. What factors might account for this difference across datasets? 2.A simple analysis of the learned gating values, such as their average magnitude on a representative dataset, would provide further insight into how the mechanism operates. 3.Please provide a brief analysis of the learned gating values, such as their average magnitude on a representative dataset, to illustrate how the model uses cross-view guidance. 4.The practical computational overhead deserves further discussion. Please report the parameter count and training time of HRG-MVC compared with the residual encoder using mean-pooling consensus. 5.The theoretical results in Appendix A provide useful insights into residual learning and cross-view guidance. However, Proposition A.1 relies on a bounded Jacobian assumption, whereas Proposition A.2 considers a scalar gating coefficient. It would be useful to clarify how these assumptions relate to the practical implementation of HRG-MVC.

1 comment
AuthorsOpenReview ↗

Response to Reviewer JAJb

Response to Question 1:

Thank you for your comment. The original Table 4 used a narrower hyperparameter search space than the main experiments. We therefore repeated the ablations, independently re-tuning each variant under the same search space and budget, and added UCI-Digit.

Table 1. Ablation of cross-view guidance and gating (%).

DatasetVariantACCNMIARI
Caltech-5VNo guidance91.7184.5183.22
Ungated90.7983.7781.72
Gated (Ours)92.0784.7083.53
HandwrittenNo guidance98.2595.8696.15
Ungated97.7594.5995.06
Gated (Ours)98.4596.2996.60
UCI-DigitNo guidance87.4087.7482.63
Ungated97.1593.3193.78
Gated (Ours)97.4093.9094.32

Gated guidance improves ACC by 0.36 and 0.20 points over no guidance on Caltech-5V and Handwritten, respectively. The smaller gain on Handwritten likely reflects its near-saturated performance, whereas Caltech-5V may offer greater cross-view complementarity.

Importantly, ungated guidance degrades performance on both datasets, potentially due to indiscriminate injection of redundant or inconsistent information. Feature-wise gating selectively controls cross-view contributions, improving all three metrics. On UCI-Digit, ungated guidance is already beneficial, but gating further improves performance, confirming its effectiveness across different datasets.

Response to Questions 2 and 3:

Thank you for your suggestions. We analyzed the learned gates and relative cross-view correction magnitudes on Caltech-5V and Handwritten using the final model (seed 5, epoch 600), averaged over samples, views, and features.

Table 2. Learned gate statistics and cross-view corrections.

DatasetLevelGate mean ± SDRelative correction norm
Caltech-5V20.4498 ± 0.48174.5115
30.4396 ± 0.40720.1229
Handwritten20.4341 ± 0.37401.6447
30.1997 ± 0.25310.2422

Note: Relative correction norm measures the L2 norm of the gated correction relative to the local representation. SD reflects gate-value dispersion.

The results reveal adaptive, level-dependent cross-view guidance. Gate means remain similar across levels on Caltech-5V but decrease substantially on Handwritten (0.4341 to 0.1997). Meanwhile, relative correction norms decrease markedly at deeper levels on both datasets, suggesting that gating allows stronger relative corrections at earlier stages while preserving deeper representations.

Response to Question 4:

Thank you for your suggestion. We compared HRG-MVC with Residual + Mean, which replaces the Cross-View Transformer with mean pooling while retaining the residual backbone.

Table 3. Computational cost and clustering performance.

DatasetMethodParams (M)Training (s/epoch)ACC (%)
Caltech-5VResidual + Mean48.7270.838280.07
HRG-MVC58.8890.913492.07
HandwrittenResidual + Mean54.6590.875597.80
HRG-MVC64.8230.909798.45
ALOI-100Residual + Mean39.1703.093683.22
HRG-MVC49.7263.161188.73

The Transformer introduces modest computational overhead, increasing training time by 0.075, 0.034, and 0.068 seconds per epoch while improving ACC by 12.00, 0.65, and 5.51 points, respectively, demonstrating a favorable performance–cost trade-off.

Response to Question 5:

Thank you for your comment. Both propositions establish sufficient theoretical conditions rather than constraints explicitly enforced during training. Proposition A.1 uses a bounded Jacobian assumption to characterize local residual stability. Although this bound is not enforced, additive shortcuts provide a direct information pathway that helps mitigate depth-induced degradation.

Proposition A.2 adopts scalar gating for analytical simplicity, whereas HRG-MVC employs data-dependent feature-wise sigmoid gating, allowing different features to receive different correction strengths. Thus, the propositions explain the motivation behind our design without guaranteeing that their conditions always hold in practice. We will clarify these connections in the revision.

Reviewer_JQw9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a hierarchical residual learning framework HRG-MVC for deep multi-view clustering, which uses residual connections to preserve useful representations across encoder layers and introduces a cross-view transformer to provide additional global guidance. A feature-wise gating mechanism was also designed to control the absorbability of cross-view information to deifferent layers.

Critical strengths

The hierarchical residual design is well motivated and provides an effective way to preserve useful information across different encoder layers. The experimental evaluation is comprehensive, and the ablation studies provide useful insights into the contributions of different components.

Critical weaknesses
  1. Most experimental results are obtained using a fixed random seed.
  2. Some design choices, particularly the number of hierarchical levels and the final prediction aggregation strategy, could be explained more clearly.
Questions for authors
  1. Could the authors provide the mean accuracy over three independent runs on one or two representative datasets?
  2. Please explain why three hierarchical levels are adopted in the main experiments?
  3. I suggest clarifying the motivation for using equal-weight averaging rather than relying only on the final-level prediction.
  4. The sensitivity analysis results show noticeable performance variations. The authors should briefly discuss the sensitivity of HRG-MVC to these hyperparameters.
1 comment
AuthorsOpenReview ↗

Response to Reviewer JQw9

Response to Question 1:

Thank you for your suggestion. We conducted additional experiments with three random seeds, independently re-tuning hyperparameters for each. Performance remained generally comparable across seeds after tuning. In the main experiments, all methods used the same fixed seed with independent hyperparameter optimization for consistent comparison. We therefore retain this protocol without claiming statistical significance.

Response to Question 2:

Thank you for your comment. To further investigate this observation, we extended the depth ablation from 3, 5, and 7 layers to 1–7 layers. The results are presented below.

Table 1. Depth ablation results (ACC, %; No / Yes residual).

DepthCaltech-5VHandwrittenALOI-100
1— / —21.85 / 19.502.70 / 2.00
287.79 / 88.4397.65 / 97.8586.56 / 88.79
388.64 / 92.0797.95 / 98.4582.45 / 88.73
469.79 / 85.7197.75 / 97.8554.77 / 73.88
561.36 / 86.7997.30 / 97.8050.36 / 82.28
664.57 / 86.5794.55 / 96.9064.84 / 79.32
728.50 / 85.8692.60 / 96.8566.99 / 84.74

Note: No / Yes indicates without / with residual shortcuts, respectively.

The results show that clustering performance generally improves at moderate depths but declines when the network becomes excessively deep. This is likely because additional layers introduce redundant transformations and optimization difficulties, potentially weakening discriminative cluster structures. Residual connections alleviate, rather than completely eliminate, such depth-induced degradation by preserving information across successive transformations. Notably, the three-layer model achieves strong performance across all three datasets, indicating that three hierarchical levels provide a favorable balance between representation capability, clustering performance, and computational cost. We will clarify this observation and the choice of three levels in the revised manuscript.

Response to Question 3:

Thank you for your comment. Equal-weight averaging is motivated by the complementary clustering information captured at different hierarchical levels. Earlier levels preserve relatively fine-grained information, while deeper levels learn more abstract representations. Relying solely on the final level may discard useful intermediate information, particularly when deeper representations suffer from information degradation. Moreover, our inter-level cluster contrastive learning encourages consistent cluster assignments across levels, making their aggregation meaningful. Equal weighting provides a simple, parameter-free way to integrate these complementary predictions, avoiding additional weighting parameters or label-dependent selection. We will clarify this motivation in the revised manuscript.

Response to Question 4:

Thank you for your comment. As shown in Figure E.3, HRG-MVC exhibits some sensitivity to the two contrastive loss weights, particularly under extreme parameter settings. Both datasets generally favor relatively small values of λ1\lambda_1, while the preferred λ2\lambda_2 differs. This is because excessively large contrastive weights may disrupt the balance between reconstruction and contrastive learning, leading to degraded representations. Nevertheless, strong performance can be achieved within reasonable parameter ranges, indicating that the method does not rely on a single isolated configuration. We will further clarify these observations and emphasize the importance of balancing the loss components in the revised manuscript.

Reviewer_GNvVOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
1
Read review
Paper summary

Aiming at the problem that stacked nonlinear encoding in deep multi-view clustering weakens the useful structure in the early layers, this paper proposes HRG-MVC, each layer retains the current input through a projected additive shortcut, and the nonlinear branch only learns the increment. Then, a single-layer Cross-View Transformer estimates hierarchical consensus between views of the same sample, and writes back to the next layer's residual through feature dimension gating. Experiments on six benchmarks demonstrate that the proposed method is more stable than existing methods in deep multi-view clustering.

Critical strengths

The main advantage of this paper may be that the research motivation is clear, the authors clearly explain the research problem, and design some experiments to verify the effectiveness of the proposed method.

Critical weaknesses

1.The primary issue of this paper is the lack of novelty, the research problem is not interesting enough, the techniques used are a combination of existing methods, and the rationality of this combination has not been fully demonstrated. The main contribution of this paper is to propose a residual structure to alleviate the performance degradation problem of deep networks, but whether it is necessary to use more layers in practical applications still needs further discussion. 2. The experimental design of this paper has some problems, and the stability and significance of the experimental results have not been fully verified. The ablation experiments were only conducted on two datasets, and on one of the datasets, the performance actually improved after removing the module, which may indicate that the module is not important on that dataset. The hyperparameter selection in this paper relies on true labels, which may affect the performance of the method in unsupervised practical applications. In addition, this paper lacks an analysis of the actual running time.

Questions for authors
  1. I understand that the authors show in Table 3 that the proposed method can alleviate the performance degradation caused by deepening the network layers. Since deepening the network layers leads to a decrease in clustering performance and an increase in computational cost, why is it necessary to have more layers? The main contribution of this work is to propose a residual structure to alleviate the performance degradation problem of deep networks, but whether it is necessary to use more layers in practical applications still needs further discussion. In addition, the proposed method is a combination of existing methods and lacks novelty.
  2. The authors conducted ablation experiments on residual shortcuts and cross-view guidance, but it seems that the ablation results of cross-view guidance show a greater decrease in performance on Caltech-5V than residual shortcuts. Does this mean that cross-view guidance is more important than residual shortcuts?
  3. Although the authors provided grid search results for hyperparameters, the hyperparameter selection in this paper relies on true labels. Does this mean that the proposed method may not achieve optimal performance in unsupervised practical applications? The authors should consider providing a more practical hyperparameter selection method that does not rely on true labels.
  4. All experiments are recommended to be conducted with multiple random seeds, reporting the mean and standard deviation to verify the stability of the method. It is also recommended to conduct significance tests to verify the effectiveness of the method.
  5. Although the authors conducted a theoretical analysis of computational complexity, they did not analyze the actual running time. It is recommended that the authors include a comparison of actual running time in the experiments to verify the efficiency of the method.
  6. Typos: In the Figure 1, the feature dimension should perhaps be dvd_v instead of vdvd. In line 269, 'Algorithm A.1' should be 'Algorithm B.1'.
3 comments
AuthorsOpenReview ↗

Response to Reviewer GNvV

Response to Question 1:

Thank you for your comment. Multiple layers enable progressive feature refinement and cross-level interaction. To examine their necessity, we extended the depth ablation from 3, 5, and 7 layers to 1–7 layers.

Table 1. Depth ablation (ACC, %; No / Yes residual).

DepthCaltech-5VHandwrittenALOI-100
1— / —21.85 / 19.502.70 / 2.00
287.79 / 88.4397.65 / 97.8586.56 / 88.79
388.64 / 92.0797.95 / 98.4582.45 / 88.73
469.79 / 85.7197.75 / 97.8554.77 / 73.88
561.36 / 86.7997.30 / 97.8050.36 / 82.28
664.57 / 86.5794.55 / 96.9064.84 / 79.32
728.50 / 85.8692.60 / 96.8566.99 / 84.74

The results show that three layers provide a favorable performance–complexity trade-off, while excessive depth may cause information degradation. Residual shortcuts substantially mitigate this issue: from 3 to 7 layers, ACC drops by 60.14/15.46 points without residuals on Caltech-5V/ALOI-100, versus only 6.21/3.99 points with residuals.

Regarding novelty, our primary contribution is hierarchical residual learning tailored to deep multi-view clustering, which preserves view-specific information and mitigates depth-induced degradation. Unlike conventional residual learning, our framework incorporates progressive, gated cross-view guidance to complement and refine residual representations, enabling stable hierarchical learning with adaptive cross-view enhancement.

Response to Question 2:

Thank you for your comment. The original ablations used a narrower hyperparameter search space. We repeated them with independent re-tuning under the same search space and budget as the main experiments, and added UCI-Digit.

Table 2. Ablation of cross-view guidance and gating (%).

DatasetVariantACCNMIARI
Caltech-5VNo guidance91.7184.5183.22
Ungated90.7983.7781.72
Gated (Ours)92.0784.7083.53
HandwrittenNo guidance98.2595.8696.15
Ungated97.7594.5995.06
Gated (Ours)98.4596.2996.60
UCI-DigitNo guidance87.4087.7482.63
Ungated97.1593.3193.78
Gated (Ours)97.4093.9094.32

Guidance is not necessarily more important than residual learning. On Caltech-5V, removing guidance reduces ACC by only 0.36 points, versus 3.43 points when removing residuals (Table 1).

Ungated guidance degrades performance on Caltech-5V and Handwritten, potentially due to indiscriminate consensus injection. Feature-wise gating adaptively controls cross-view contributions, improving all metrics. On UCI-Digit, ungated guidance is already beneficial, but gating further improves performance. These results confirm the complementary roles of residual learning, guidance, and gating.

Response to Question 3:

Thank you for your comment. Benchmark-based tuning differs from unsupervised training and inference. Labels are used only during model development to identify suitable configurations for subsequent unlabeled tasks, never during training or prediction. In practice, clustering stability across random initializations or data perturbations offers a potential label-free hyperparameter selection criterion. We will clarify this distinction in the revision.

Response to Question 4:

Thank you for your suggestion. We conducted additional experiments with three random seeds, independently re-tuning hyperparameters for each. Performance remained generally comparable across seeds after tuning. In the main experiments, all methods used the same fixed seed with independent hyperparameter optimization for consistent comparison. We therefore retain this protocol without claiming statistical significance.

Response to Question 5:

Thank you for your suggestion. We compared HRG-MVC with Residual + Mean, which replaces the Cross-View Transformer with mean pooling.

Table 3. Computational cost and clustering performance.

DatasetMethodParams (M)Training (s/epoch)ACC (%)
Caltech-5VResidual + Mean48.7270.838280.07
HRG-MVC58.8890.913492.07
HandwrittenResidual + Mean54.6590.875597.80
HRG-MVC64.8230.909798.45
ALOI-100Residual + Mean39.1703.093683.22
HRG-MVC49.7263.161188.73

The Transformer adds only 0.075, 0.034, and 0.068 seconds per training epoch, while improving ACC by 12.00, 0.65, and 5.51 percentage points, respectively, demonstrating a favorable efficiency–performance trade-off.

Response to Question 6:

Thank you for pointing out these typos. We have corrected the feature dimension notation in Figure 1 and the algorithm reference from “Algorithm A.1” to “Algorithm B.1”. We appreciate your careful reading.

The authors addressed the questions I raised satisfactorily, and I raise my score to 3.

Dear Reviewers and Authors,

As a quick reminder, the author-reviewer discussion period will close on November 18.

@Reviewers: Please make sure to read the authors' rebuttals, acknowledge their responses, and engage in further discussion if any points remain unclear. While final decisions are not made at this stage, we ask that you carefully re-evaluate your ratings in light of the new clarifications and experimental results:

Increase your score if the authors have satisfactorily resolved your concerns and strengthened the paper.

Decrease your score if new, unaddressed issues have emerged during the discussion.

Maintain your score if critical concerns remain unresolved.

Please note that you are under no obligation to change your rating, provided your final score accurately reflects the current state of the manuscript. To ensure a fair evaluation, please ground your assessment in concrete evidence rather than personal preference, and consider whether the work offers meaningful scientific value to the ICLR community.

Thank you very much for your time, thoughtful engagement, and contributions to the review process.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.