acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Modality Conflicts in Multi-modal Clustering with Decoupled Neighborhoods

Abstract

Multi-modal clustering aims to discover underlying cluster structures by exploring information from different modalities. However, existing methods often struggle with conflicts across diverse modalities, where modality-specific information often leads to inconsistent topological relationships. To address this issue, we propose a novel neighborhood decoupling strategy that decomposes multi-modal graphs into a set of edge-disjoint graphs based on cross-modal edge frequencies. This is driven by the insight that edges repeatedly supported by multiple modalities exhibit higher semantic reliability. By stratifying the structural information into a hierarchy of decoupled neighborhoods according to edge frequencies, we introduce a structure alignment objective that enforces a structural ordering and injects the hierarchical topology into the modality-joint representation for clustering. Furthermore, we establish theoretical guarantees showing that optimizing this objective minimizes an upper bound on the -means cost, ensuring compact cluster structures. Extensive experiments are conducted to verify the superiority of the proposed method compared with state-of-the-art approaches on multiple benchmark datasets.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multi-modal clustering, Decoupled Neighborhoods, Modality Conflicts

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FQRpOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Decoupled Neighborhood-Induced Multi-Modal Clustering (DNI-MMC) for mitigating modality conflicts in multi-modal clustering. The key idea is to decouple multi-modal graphs into multiple edge-disjoint frequency-aware graphs according to the cross-modal occurrence frequency of edges. High-frequency edges are treated as more reliable and are expected to capture shared consensus, while lower-frequency edges are treated as auxiliary or less reliable information. Based on this decoupling, the paper introduces a structure alignment loss and a tractable surrogate objective consisting of a soft ranking loss, a variance penalty, and a negative constraint. Theoretical results include an upper bound relating the alignment loss to the surrogate loss, a k-means cost bound, and a mini-batch variant. Experiments are conducted on five multi-modal datasets and several multi-modal attributed graph datasets, with comparisons against a number of shallow and deep baselines.

Critical strengths

The paper proposes a simple and intuitive frequency-based neighborhood decoupling strategy to mitigate modality conflicts in multi-modal clustering.

The structure alignment loss and its surrogate ranking-based formulation provide a reasonable way to inject hierarchical neighborhood information into the modality-joint representation.

The paper provides theoretical analyses, including an upper bound for the alignment loss and a k-means cost bound.

The method is evaluated on both multi-modal clustering and multi-modal attributed graph clustering tasks, showing competitive performance on several benchmarks.

Ablation studies, hyperparameter sensitivity analysis, and visualization are provided.

Critical weaknesses

The assumption that cross-modal edge frequency is a reliable proxy for semantic consistency is not sufficiently validated. The paper only states that statistical analysis reveals a strong positive correlation between edge frequency and the homophily ratio, but does not report correlation coefficients, significance tests, or per-dataset details. High-frequency edges may also arise from shared noise or common biases across modalities, rather than true semantic agreement.

The proof of Theorem1 appears to contain formatting and derivation issues. For example, in Case2, the expression

2βi24(μu(0))22\beta_i^2 4(\mu_u^{(0)})^2
seems to be missing an operator, and the constants in the final bound need to be carefully checked against the definition of $\mathcal{L}_{tra}$. The statement
Lalig≤Ltra+2∑i=1ℓβi2volA(i)\mathcal{L}_{alig} \leq \mathcal{L}_{tra} + 2\sum_{i=1}^{\ell}\beta_i^2 \mathrm{vol}_{\mathbf{A}^{(i)}}
should be verified with all coefficients and assumptions explicitly stated.

Theorem2 and Corollary1 depend on the true cluster indicator vector y\mathbf{y}, which is unavailable in unsupervised clustering. This is acceptable as an oracle-style theoretical analysis, but the paper should clearly state this limitation and avoid implying that the bound is directly computable or optimizable in practice.

The experimental tables are poorly formatted and difficult to verify. In Table~1, method names and numerical values are concatenated (e.g., ``AWPEEMVCCAMVC''), several columns appear missing, and the reported improvements are not always consistent with the claim of consistent superiority. For instance, the NMI improvement on nuswide is negative (−0.04-0.04), and the ARI improvement on Photo is also negative (−0.27-0.27). The paper should report complete results, standard deviations, and statistical significance tests.

The paper claims state-of-the-art performance, but several improvements are very small. For example, on Reuters, the ACC improvement is only 0.020.02, and on nuswide, the NMI is slightly worse than the best baseline. The claims should be softened or supported by significance testing.

Hyperparameter and implementation details are insufficient. The sequence {βi}i=1ℓ\{\beta_i\}_{i=1}^{\ell} is only constrained by 0<β1≤β2≤⋯≤βℓ≤10 < \beta_1 \leq \beta_2 \leq \cdots \leq \beta_{\ell} \leq 1, but the actual values or learning strategy are not specified. The learning rate is written as ``-01'', which is clearly a typo. The choice of 2,500 training epochs, early stopping, batch size, network architecture, loss weights, random seeds, and number of repeated runs are not adequately described.

The scalability and computational complexity of the proposed method are not sufficiently analyzed. The negative term in Lalig\mathcal{L}_{alig} sums over Nu(0)\mathcal{N}_u^{(0)}, i.e., samples outside the nearest neighborhoods across all modalities, which can be O(n2)O(n^2) in a naive implementation. Although mini-batch training is mentioned, no complexity analysis or large-scale experiments are provided. The GPU memory entry in Table~6, ``2552.47 for N(0)'', is unclear.

The ablation study in Table~2 contains duplicated or mislabeled rows, such as ``Lrank + Lreg'' appearing twice. It is therefore difficult to determine the individual contributions of Lrank\mathcal{L}_{rank}, Lvar\mathcal{L}_{var}, and Lneg\mathcal{L}_{neg}. A corrected and more detailed ablation is needed.

The multi-modal attributed graph clustering section does not clearly explain how the graph edges E\mathcal{E} are combined with the modality-specific graphs constructed from attributes. The statement ``obtain the decoupled neighborhoods from them and E\mathcal{E}'' lacks algorithmic detail. In addition, the dataset statistics in Table5 are inconsistent with the main text: Grocery-S is used in experiments but does not appear in Table5.

The paper positions its frequency-based decoupling as fundamentally different from binary consistency-specificity decomposition, but frequency stratification is still a heuristic. Low-frequency edges may contain useful complementary information, and the ranking-based surrogate may overly suppress them. More analysis is needed to justify why the proposed hierarchy is preferable.

Questions for authors

Can you provide quantitative statistics for the relationship between edge frequency and edge homophily ratio, including correlation coefficients, significance tests, and per-dataset results?

Can you correct or clarify the proof of Theorem1 and Theorem2, including all constants, assumptions, and the precise definition of Ltra\mathcal{L}_{tra}? How should the dependence on the true cluster labels y\mathbf{y} be interpreted in an unsupervised setting?

Can you provide complete and correctly formatted tables for all experiments, including all baseline results, standard deviations, and statistical significance tests?

How are the weights {βi}i=1ℓ\{\beta_i\}_{i=1}^{\ell} set in practice? How are the loss weights, learning rate, batch size, network architecture, and number of training epochs selected? Is early stopping used?

Can you provide a computational complexity analysis of the proposed method and scalability experiments on larger datasets?

Can you correct the ablation study in Table~2 and clearly report the individual effects of Lrank\mathcal{L}_{rank}, Lvar\mathcal{L}_{var}, and Lneg\mathcal{L}_{neg}?

In the multi-modal attributed graph clustering task, how are the modality-specific graphs constructed and combined with the given graph edges E\mathcal{E}? Why is Grocery-S missing from Table~5?

Will code and preprocessing scripts be released to support reproducibility?

4 comments

We thank the reviewer for their positive and supportive assessment of our work, as well as their constructive feedback. We address the concerns below.

1. Quantitative evidence linking edge frequency to edge homophily

Our paper reports frequency-specific edge homophily on all five multimodal datasets: CUB, YouTube, NUS-WIDE, AWA, and Reuters in Figure 1 (b). In this part, we add the Pearson correlations between edge-frequency levels and their corresponding homophily ratios in the following table.

DatasetPearson rp-value
YouTube0.95810.00260
Reuters0.93010.00775

These observations support edge frequency as an empirical prior for average semantic reliability for the motivation of neighborhood decoupling.

2. Theorem 1 and the role of ground-truth labels

We have rechecked the proof of Theorem 1 in our manuscript and confirmed that Case 2 already includes the addition operator and that the constants associated with LtraL_{tra} are correct. Specifically, LtraL_{tra} consists of three components, each derived from the upper bound on the alignment loss. Our paper discusses these components immediately after introducing LtraL_{tra}.

Our theoretical analysis evaluates the within-cluster compactness of the learned representations using the k-means cost with respect to the ground-truth partition. To further characterize the quality of the multi-modal data, we introduce the edge homophily ratio, which measures the semantic consistency of graph edges with the ground-truth clusters. These quantities are used solely for theoretical analysis to connect the optimization objective with within-cluster compactness. Specifically, our results upper-bound the k-means cost in terms of the training objective and the edge homophily ratios of individual modalities. Ground-truth labels are not required to train or apply our method.

3. Experimental tables

Upon acceptance, we will improve column spacing, separate method names clearly, and include the corresponding standard deviations and statistical significance tests in the final manuscript.

4. Practical treatment of β\beta and implementation details

An important clarification is that our proposed surrogate loss does not require explicitly setting or learning β\beta. These weights appear in the idealized alignment objective and its theoretical analysis, whereas the ranking surrogate replaces exact similarity targets with ordinal constraints.

In the ablation study, we implement a learnable version of β\beta in LaligL_{\mathrm{alig}} by introducing trainable parameters and applying the softplus function to ensure positivity and range constraints. As discussed in our paper, β\beta is related to the edge homophily ratio and is difficult to determine reliably without ground-truth cluster labels. The ablation results show that LaligL_{\mathrm{alig}} with learnable β\beta yields worse clustering performance than our proposed surrogate loss, supporting the effectiveness of the surrogate formulation.

We use a learning rate of 10−410^{-4}, a batch size of 4096, and 2500 training epochs, without early stopping. All modality-specific encoders use fully connected architectures, with layer dimensions DiD_i–196–128–64 and 64–128–196–DiD_i, respectively, where DiD_i denotes the feature dimension of the ii-th modality. The feature fusion layer is implemented using a single MLP.

5. Computational complexity and scalability

Our method uses mini-batch training to handle large-scale datasets. Let bb denote the batch size, DD the sum of the input dimensions across modalities, and dd the dimension of the fused representation. With brute-force exact neighborhood construction, computing pairwise distances takes O(b2D)O(b^2D) time, with an additional cost for neighbor selection. Computing fused similarities takes O(b2d)O(b^2d) time, while evaluating the proposed surrogate loss takes O(ℓ2b2)O(\ell^2b^2) time, where ℓ\ell is the number of modalities. The combined computational complexity of these components per batch is therefore O(ℓ2b2+b2D+b2d)O(\ell^2b^2 + b^2D + b^2d).

Our experiments include five datasets with more than 10,000 samples: AWA, Reuters, Reddit-S, Photo, and Grocery-S. The results on these datasets provide empirical support for the scalability of our method.

6. Ablation study and the contribution of each loss term

We rechecked Table 2 and did not find duplicated rows in the manuscript version we checked. Nevertheless, we will improve the labels and formatting to make each configuration unambiguous.

To study the individual effects of each loss term, we add a baseline for Lrank+LvarL_{rank}+L_{var} and report the clustering performance in the following table.

DatasetObjectiveACCNMIARIF-score
YouTubeLrank+LnegL_{rank}+L_{neg}59.9045.1638.5145.50
YouTubeLvar+LnegL_{var}+L_{neg}43.8630.3522.5431.52
YouTubeLrank+LvarL_{rank}+L_{var}45.2830.2922.9730.73
YouTubeFull objective61.3145.5739.5546.27
AWALrank+LnegL_{rank}+L_{neg}68.7475.9160.5361.63
AWALvar+LnegL_{var}+L_{neg}67.7174.6659.7160.37
AWALrank+LvarL_{rank}+L_{var}40.3657.3631.8633.70
AWAFull objective69.5175.9961.9161.81
ReutersLrank+LnegL_{rank}+L_{neg}53.4835.3728.2445.75
ReutersLvar+LnegL_{var}+L_{neg}46.2828.1120.4141.39
ReutersLrank+LvarL_{rank}+L_{var}54.9536.6330.0644.65
ReutersFull objective55.8636.9731.0546.39

The results demonstrate the effectiveness of LnegL_{neg}, LrankL_{rank} and LvarL_{var}, supporting the use of the full objective Lrank+Lvar+LnegL_{rank} + L_{var} + L_{neg}.

7. Details for Multi-modal attributed graph clustering

In our paper, we first use heat kernel diffusion on the original graph to incorporate topological information into the attributes of both modalities (image and text). We then construct a modality-specific kNN graph from the diffused attributes of each modality and use the resulting two graphs to construct the decoupled neighborhood graphs.

Thank you for pointing out the omission of Grocery-S from Table 5. We will add the missing entry to Table 5 and correct the table in the final manuscript upon acceptance.

8. Reproducibility

We will release the implementation and data preprocessing scripts upon acceptance of our paper.

9. Why use a frequency hierarchy, and how are low-frequency edges treated?

We agree that frequency stratification is a heuristic informed by empirical observations. Our claim is that it provides a useful way to represent differences in average reliability across neighborhoods. The distinction from binary consistency–specificity decomposition is that our method retains multiple frequency levels and models their relative relationships. Low-frequency neighborhoods remain part of the learning objective, allowing their potentially useful complementary information to contribute. More specifically, the ranking surrogate constrains average similarities across frequency-specific neighborhoods. It does not require every high-frequency pair to have greater similarity than every low-frequency pair, nor does it treat all low-frequency edges as negative pairs. Individual relations can therefore deviate from the average ordering. This design provides flexibility to retain useful low-frequency relations.

The manuscript has been substantially improved after revision and, in my view, now meets the standard of ICLR. However, to ensure transparency and reproducibility, the authors must add the public code link as the final sentence of the Abstract. In addition, the authors must ensure that the released code is complete and runnable, including all necessary scripts/instructions to reproduce the reported experiments and figures.

Thank you for taking the time to review the response and for your positive assessment. We appreciate your suggestions for improving transparency and reproducibility. We will release the complete, runnable code in a public repository and add the repository link as the final sentence of the abstract upon acceptance of our paper.

Reviewer_Bb4uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This manuscript proposes DNI-MMC to address modality conflicts arising from structural inconsistencies among modality-specific neighborhood graphs. The method partitions edges into disjoint frequency-based layers according to their occurrence across modalities and learns fused representations through ranking, variance, and non-neighbor constraints. The theoretical analysis relates the training loss to an upper bound on the within-cluster cost. Experiments on five multimodal datasets and three attributed graph datasets show improvements over the compared baselines on most metrics. The overall approach is clear, but the methodological contribution and supporting evidence require further strengthening. First, the manuscript does not specify the number of repeated runs or report variability across runs, leaving the statistical reliability of some smaller gains unclear. Second, the theoretical bound involves homophily and cluster sizes that depend on ground-truth labels. Its tightness and practical implications for optimization remain unverified. Third, although the appendix compares coarse- and fine-grained stratification, three of the five standard multimodal datasets contain only two modalities, limiting the evidence for the benefits of finer frequency-based decomposition. Regarding novelty, the authors should more clearly articulate the substantive differences between the proposed frequency-based stratification and ranking objective and existing consistency-modeling approaches. They should also examine whether edge frequency remains a reliable proxy for semantic agreement in the presence of noisy or redundant modalities. Overall, the method demonstrates empirical promise, but the distinctiveness of its core mechanism and the conditions under which it is effective require stronger justification.

Critical strengths
  1. The paper addresses structural inconsistencies among modality-specific neighborhood graphs, a relevant issue in multimodal clustering. The motivation is clearly presented, and the proposed neighborhood decomposition is reasonably aligned with the identified problem. 2.Organizing neighborhood relations by cross-modal edge frequency provides an intuitive and interpretable way to exploit structural agreement. The ranking constraints establish a clear connection between these frequency levels and the similarities learned in the fused representation. 3.The paper provides theoretical analysis linking the training objective to an upper bound on the within-cluster cost. Although the practical informativeness of this bound requires further investigation, the analysis offers some theoretical support beyond an exclusively empirical justification. 4.The experiments cover both conventional multimodal clustering and multimodal attributed graph clustering, with competitive results on most reported metrics. Component ablations, sensitivity analysis, and comparisons between coarse- and fine-grained stratification provide additional evidence for the design choices.
Critical weaknesses
  1. The method treats edges recurring across modalities as more reliable neighborhood relations. However, modalities may share noise or be highly redundant, in which case high-frequency edges do not necessarily connect samples from the same class. Although the paper provides empirical support for this assumption, its validity under noisy, redundant, or uneven-quality modalities has not been systematically established.
  2. The edge-frequency computation assigns equal contributions to all modalities, regardless of their quality or informational independence. Consequently, a low-quality modality contributes equally to a high-quality one, while correlated modalities may amplify the same potentially misleading relations. The paper would benefit from clarifying the conditions under which this equal-weight treatment is appropriate or evaluating its sensitivity through controlled experiments.
  3. Three of the five conventional multimodal datasets—CUB, nuswide, and AWA—contain only two modalities and therefore yield only two nonzero frequency levels. Improvements on these datasets alone do not establish the benefits of finer-grained stratification. Although the appendix compares coarse- and fine-grained grouping on Youtube and Reuters, further analysis of how the benefits depend on the number of modalities and frequency levels would strengthen the evidence.
  4. The manuscript states that experiments are repeated and mean results are reported, but it does not specify the number of runs or provide standard deviations, confidence intervals, or significance analyses. For relatively small performance differences, it is therefore difficult to determine whether the reported gains exceed the variability induced by random initialization and clustering.
  5. The theoretical bound involves homophily and cluster sizes that depend on unavailable ground-truth labels, and its tightness or non-vacuity is not demonstrated. Reducing a loss-dependent term in an upper bound does not necessarily imply a reduction in the actual within-cluster cost or clustering error. The practical implications of the theoretical results for optimization and model selection therefore require further clarification.
  6. The ablation results do not fully support the claim that the complete objective consistently achieves the best performance. In Table 2, the ranking loss alone yields an ARI of 35.15 on Reuters, compared with 31.05 for the full objective. Moreover, the (L_{\mathrm{rank}}+L_{mathrm{var}}) configuration is absent, preventing a direct assessment of the incremental contribution of the negative constraint when both other components are retained.
  7. Neighborhood graphs are reconstructed within mini-batches, so both neighborhood membership and cross-modal edge frequency may vary with batch composition. Although the paper provides related theoretical analysis, the empirical effects of batch size and sampling strategy on the stability of frequency-based stratification and final clustering performance remain insufficiently explored.
Questions for authors
  1. When modalities exhibit shared noise, substantial redundancy, or differences in quality, do higher-frequency edges still exhibit higher homophily? Could the authors provide controlled experiments to clarify when this assumption holds and identify potential failure cases?
  2. What motivates assigning equal contributions to all modalities when computing edge frequency? Could highly correlated or low-quality modalities lead to redundant evidence being counted multiple times or unreliable relations being amplified?
  3. Under what conditions is the theoretical upper bound non-vacuous and reasonably tight? Given its dependence on ground-truth homophily and cluster sizes, could the authors clarify how it informs practical training, or empirically examine the relationship between decreasing training loss and changes in the within-cluster cost?
  4. In Table 2, the ranking loss alone achieves a higher ARI on Reuters than the complete objective. How do the authors explain this result? Could they also include the Lrank+LvarL_{\mathrm{rank}}+L_{\mathrm{var}} configuration to isolate the incremental contribution of the negative constraint?
  5. Since neighborhood membership and edge-frequency levels may change with mini-batch composition, how do the authors assess the impact of this variability on learning? Could they report the stability of the stratification and the resulting clustering performance under different batch sizes or sampling strategies?
4 comments

We thank the reviewer for the detailed and constructive feedback. Below, we address each concern.

1. Reliability of edge frequency under shared noise, redundancy, and unequal modality quality (W1, Q1)

Our method uses cross-modal edge frequency as a proxy for neighborhood reliability, motivated by the intuition that relations supported by multiple modalities generally provide stronger evidence than those observed in a single modality. As the reviewer points out, modalities may share noise or contain highly redundant information, so high-frequency edges do not necessarily connect samples from the same class. Our method accommodates such exceptions by constraining only the average similarities of frequency-specific neighborhoods, rather than requiring every higher-frequency pair to have greater similarity than every lower-frequency pair. This aggregate formulation relaxes the reliance on the reliability of individual edges and can help mitigate the influence of incorrectly connected pairs.

To characterize inter-modal disagreement, our paper introduces the modality conflict rate (MCR), defined as the proportion of samples with inconsistent clustering assignments across modalities. Figure 3 visualizes the pairwise MCR values, revealing varying levels of agreement between modalities. Although MCR does not directly measure shared noise, redundancy, or modality quality, it provides useful context for interpreting their clustering consistency. As shown in Table 1, our method outperforms competing approaches, supporting its effectiveness in handling heterogeneous modality relationships.

2. Motivation and limitations of equal contributions to edge frequency (W2, Q2)

Equal counting provides a simple, label-free measure of cross-modal support. Each modality contributes a binary indicator of whether an edge belongs to its neighborhood graph, avoiding the need to estimate modality quality or calibrate similarity magnitudes across modalities.

This design is most appropriate when the modalities provide reasonably informative and complementary neighborhood evidence. We agree that it can overcount highly correlated evidence or amplify unreliable relations when several modalities share the same errors. The loss function constrains the average similarities of frequency-specific neighborhoods, so it does not require every higher-frequency pair to be more similar than every lower-frequency pair. This allows individual relations to deviate from the average ordering. Thus, our method is able to eliminate the effects of unequal modality quality.

3. Benefits of neighborhood decoupling with two and multiple modalities (W3)

With two modalities, our method distinguishes edges supported by one modality, edges supported by both modalities, and pairs supported by neither. Thus, it still preserves information about cross-modal agreement beyond a binary neighbor/non-neighbor distinction.

To examine this distinction, we additionally evaluated a binary neighborhood-decoupling baseline on three attributed-graph datasets. Our method obtains higher scores on all twelve dataset–metric combinations. These results support retaining cross-modal support information rather than collapsing all observed edges into a single neighborhood category.

DatasetMethodACCNMIF-scoreARI
Grocery-SBinary neighborhood decoupling46.6947.3632.9338.76
Grocery-SOurs47.3248.8734.5940.12
Reddit-SBinary neighborhood decoupling79.5781.6876.2776.74
Reddit-SOurs81.0482.8977.9878.31
PhotoBinary neighborhood decoupling55.6954.2052.9728.18
PhotoOurs56.3654.2653.1728.37

As shown in Appendix B.4, Figure 6, our full stratification obtains better clustering performance than coarser grouping on YouTube (six modalities) and Reuters (four modalities). The reported ACC and NMI improvements support finer stratification in our neighborhood decoupling manner.

4. Number of repetitions and reporting of variability (W4)

All experiments were repeated 10 times, and we report the mean results across these runs. Upon acceptance, we will explicitly describe this protocol and include the corresponding standard deviations in the final manuscript.

5. Interpretation of the theoretical bound (W5, Q3)

Our theoretical analysis evaluates the within-cluster compactness of the learned representations using the k-means cost with respect to the ground-truth partition, whose labels are unavailable during training. A key insight is that the quality of the original multi-modal data affects the compactness of the learned representations. To characterize data quality, we introduce the edge homophily ratio, which measures the semantic consistency of graph edges with the ground-truth clusters.

Our theoretical results connect the optimization objective to within-cluster compactness by bounding the k-means cost in terms of the training objective and the edge homophily ratios of individual modalities. To complement this analysis, the following table empirically examines how the within-cluster cost changes as the training loss decreases. The magnitude of the cost reduction varies substantially across datasets. These observations provide empirical support for the connection between optimization and compactness.

DatasetQuantity5001000150020002500
NUS-WIDETraining loss438.55418.10408.67402.93398.92
NUS-WIDEK-means cost8004.907996.717986.977975.427967.23
AWATraining loss419.86397.86388.24383.46379.70
AWAK-means cost6543.346118.435900.965754.635648.47
ReutersTraining loss1341.781319.601303.121293.671280.39
ReutersK-means cost17449.2217448.5817432.2917425.3017424.24

6. Reuters ARI and the contribution of the negative constraint (W6, Q4)

Although the ranking-only variant achieves a higher ARI on Reuters (35.15 versus 31.05), the full objective performs better in ACC, NMI, and F-score, supporting its overall effectiveness.

To isolate the contribution of the negative constraint, we additionally evaluated Lrank+LvarL_{rank}+L_{var} and report its clustering performance in the following table.

DatasetObjectiveACCNMIARIF-score
YouTubeLrank+LnegL_{rank}+L_{neg}59.9045.1638.5145.50
YouTubeLvar+LnegL_{var}+L_{neg}43.8630.3522.5431.52
YouTubeLrank+LvarL_{rank}+L_{var}45.2830.2922.9730.73
YouTubeFull objective61.3145.5739.5546.27
AWALrank+LnegL_{rank}+L_{neg}68.7475.9160.5361.63
AWALvar+LnegL_{var}+L_{neg}67.7174.6659.7160.37
AWALrank+LvarL_{rank}+L_{var}40.3657.3631.8633.70
AWAFull objective69.5175.9961.9161.81
ReutersLrank+LnegL_{rank}+L_{neg}53.4835.3728.2445.75
ReutersLvar+LnegL_{var}+L_{neg}46.2828.1120.4141.39
ReutersLrank+LvarL_{rank}+L_{var}54.9536.6330.0644.65
ReutersFull objective55.8636.9731.0546.39

The results demonstrate the effectiveness of adding LnegL_{neg} to Lrank+LvarL_{rank} + L_{var}, supporting the use of the full objective Lrank+Lvar+LnegL_{rank} + L_{var} + L_{neg}.

7. Effects of mini-batch construction and batch size (W7, Q5)

Corollary 1 analyzes uniformly sampled mini-batches without replacement under the condition b>akb>ak, where bb is the batch size and ∣Cj∣≥n/(ak)|\mathcal{C}_j|\geq n/(ak). It relates the global within-cluster cost to an expectation over quantities computed within each batch. Increasing bb reduces the amplification factors involving 1/(b−ak)1/(b-ak). This suggests that using larger batch sizes during mini-batch training can promote more compact clusters in the learned representation. To examine the effect of batch size, we report the clustering performance of our method across different batch sizes in the following table. The results show that larger batch sizes generally lead to better clustering performance.

Batch SizeNUS-WIDE ACCNUS-WIDE NMINUS-WIDE ARINUS-WIDE FscoreAWA ACCAWA NMIAWA ARIAWA Fscore
204867.1753.4747.9053.8769.4475.8661.5161.56
409668.6455.5550.7756.1569.5175.9961.9161.81
819269.2856.0451.3656.5769.5576.0461.5761.99

Thank you for the detailed response. I maintain my recommendation of weak rejection. The additional ablation clarifies the role of the negative constraint. The training trajectories provide the requested empirical comparison between the loss and the within-cluster cost. Three concerns remain:

  1. The response clarifies the benefit of frequency-based stratification over binary neighborhood grouping. Comparisons with coarser groupings also support retaining edge-frequency information. However, the combination of frequency-based stratification and ranking is not clearly distinguished from the closest approaches to modeling cross-modal consistency. A direct comparison of how the proposed and related methods use agreement across modalities would help clarify the methodological contribution.

  2. Across all three datasets, the training loss and the within-cluster cost decrease together at the reported checkpoints. However, the response does not compare the numerical upper bound with the observed cost or identify conditions under which the bound is reasonably tight. The quantitative relationship between reductions in the loss-dependent term and the actual within-cluster cost remains unclear.

  3. Duplicating a weak modality increases its edge frequencies without adding independent evidence. Within-group similarity averaging accommodates individual errors, but the response does not establish the reliability of group ordering when multiple modalities share the same errors. The modality conflict rate cannot detect agreement on incorrect assignments. Moreover, no controlled experiment evaluates edge homophily at each frequency level as modality redundancy or shared noise increases. The claim that averaging eliminates the effects of unequal modality quality remains unsupported.

Overall, the response strengthens the evidence for several design choices. My main concerns remain unresolved regarding the distinction from prior work, the implications of the theoretical bound for optimization, and the reliability of the frequency ordering underlying the ranking objective.My assessment of the paper remains unchanged, and I will maintain my original score.

Thank you for the follow-up. We address the three remaining concerns below.

1. Distinguishing our method from approaches that model cross-modal consistency

Previous methods separate multimodal information into consistent and diverse components. Because the boundary between these components can be ambiguous, models often struggle to accurately and completely determine which information should be assigned to each component, limiting their ability to capture sufficient information from multi-modal data for effective clustering. Differently, our method decouples neighborhoods based on how often each edge appears across modalities. As a result, the information in the decoupled subset of neighborhoods exhibits diverse degrees of consistency with the cluster structures, enabling multi-modal information to be allocated to its most appropriate part. It then compares the mean embedding similarities of the resulting groups using a soft ranking loss. The methodological distinctions are this form of neighborhood decoupling and the use of cross-modal agreement as ordinal supervision over neighborhood means.

2. Numerical interpretation and tightness of the upper bound

Equation (8) defines a family of upper bounds indexed by the coefficients β\boldsymbol{\beta}. These coefficients specify target similarities for the frequency groups in the theoretical analysis. The training objective operates on the relative ordering of group means and does not fit numerical values for β\boldsymbol{\beta}. The trained model therefore does not by itself provide a numerical bound for comparison with the reported cost.

For fixed graphs and β\boldsymbol{\beta}, Theorem 2 establishes that reducing the surrogate loss lowers the upper bound. The theorem does not control the gap between this bound and the actual within-cluster cost. The reported trajectories show that the surrogate loss and the actual within-cluster cost decreased together at the reported checkpoints.

3. Reliability of frequency ordering under shared errors

Our earlier statement about the robustness of within-group averaging was too broad. When shared errors affect relatively few samples and the resulting erroneous edges make up a small fraction of each affected frequency group, averaging can limit their influence. For a fixed embedding and neighborhood group, if a fraction ε<1\varepsilon<1 of the edges is affected, the mean cosine similarity over all edges differs from the mean over the unaffected edges by at most 2ε2\varepsilon. The ordering can therefore be preserved when the gap between group means is sufficiently large.

When shared errors are widespread or concentrated in a high-frequency group, they can alter the group's composition and reverse the homophily ordering. In this regime, the ranking objective can reinforce erroneous neighborhood relations. One possible mitigation is to compare the structural similarity of the modality-specific neighborhood graphs and retain one representative modality from each group of highly redundant modalities before computing edge frequencies. This would reduce repeated counting of essentially the same neighborhood relations. We will discuss this failure mode and the potential mitigation in the limitations section.

Reviewer_KKagOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses modality conflicts in multi-modal clustering and proposes DNI-MMC: it decomposes multi-modal graphs into edge-disjoint frequency-stratified graphs according to how many modalities support each edge, designs a structure alignment loss and a tractable surrogate (rank, variance, and negative terms) to inject the hierarchical neighborhood topology into a modality-joint representation, and provides a k-means cost upper bound plus a mini-batch version, with results on five multi-modal datasets and three multi-modal attributed graph datasets.

Critical strengths

The core idea is simple and intuitively appealing: using cross-modal edge frequency as a proxy for structural reliability generalizes the usual consistent/specific binary decomposition into a finer multi-level hierarchy, offering a lightweight and easy-to-implement perspective on mitigating modality conflicts, with coverage of two task families and mostly leading results.

Critical weaknesses
  1. The method relies on a hard KNN threshold. The authors recommend a default ζ=20, meaning a sample ranked 21st by distance is directly treated as a non-neighbor, even though the neighbor/non-neighbor boundary is inherently blurry. The paper only reports a grid search over {10,...,50} and provides no analysis of soft/adaptive neighborhoods or of the misclassification rate for boundary samples; moreover, the search grids in Sec. 5.1 and Sec. 5.3 are inconsistent ({10,15,20,25,30,35,45,50} vs. {10,15,20,25,30,35,40,45}), which is not explained.

  2. The stratification signal may be weak when the number of modalities is small, since the number of decoupled graphs equals ℓ. In the multi-modal attributed graph experiments the data are two-modal, so only two frequency levels are available and the ordering constraint is limited; the paper reports results on this setting but provides no ablation or discussion of this low-modality regime.

  3. The central assumption that high edge frequency implies semantic reliability lacks empirical support. The introduction claims a statistical analysis reveals a strong positive correlation between edge frequency and homophily ratio, but no correlation coefficient, significance level, or dataset scope is reported, even though this assumption underpins the entire ordered weighting scheme.

Questions for authors
  1. How does the method perform when ℓ=2 or 3, where only a few frequency levels exist? Does it still outperform binary consistent/specific decomposition, and can you report an ablation for the two-modal attributed graph setting?

  2. How stable are the results if the hard KNN is replaced by soft or adaptive neighborhoods, and what fraction of boundary samples (distance near the ζ-th threshold) are misclassified as neighbors?

1 comment

We thank the reviewer for their assessment of the paper’s quality and constructive suggestions. We address the concerns below.

1. Hard KNN construction and neighborhood-boundary ambiguity (W1 + Q2)

We agree that hard KNN introduces a discontinuity: samples with similar distances may receive different neighborhood assignments. We use KNN to construct sparse local support graphs, without assuming that its cutoff exactly separates semantically related and unrelated samples.

An important clarification is that exclusion from one modality’s KNN graph does not necessarily make a pair a non-neighbor in our joint objective. A pair supported by any modality remains in a positive-frequency neighborhood; only pairs unsupported by all modalities enter the non-neighborhood set. Thus, complementary neighborhood evidence from other modalities can retain a relation excluded by an individual modality. Furthermore, the ranking term operates on the average similarity within each frequency-specific neighborhood. It does not require every higher-frequency pair to have greater similarity than every lower-frequency pair. This avoids imposing a strict ordering on individual edges, although uncertainty in graph construction remains. Hard KNN provides a simple, sparse, rank-based support indicator for frequency counting, avoiding the similarity calibration or additional parameters that soft or adaptive neighborhoods may require. These properties motivate our choice of hard KNN.

We also acknowledge the inconsistent search ranges in Sections 5.1 and 5.3. We will check the experiment configurations and correct or distinguish the reported ranges accordingly. The suggested parameter ζ=20\zeta=20 is an empirical default supported by the sensitivity study, rather than a universally optimal choice.

2. Effectiveness with a small number of modalities (W2 + Q1)

We agree that fewer modalities provide fewer frequency levels. Accordingly, we distinguish the effectiveness of the overall framework from the additional benefit of finer frequency stratification.

The existing experiments include three two-modal datasets: CUB, NUS-WIDE, and AWA. Our method achieves ACC scores of 89.97, 68.64, and 69.51, respectively, compared with 86.67, 68.68, and 67.16 for BMvC. These results demonstrate competitive performance in the two-modal setting.

With two modalities, our decomposition distinguishes single-supported edges, jointly supported edges, and unsupported pairs. The framework therefore remains applicable with two positive-frequency levels. With three modalities, exact-frequency stratification can additionally distinguish edges supported by two modalities from those supported by all three, which the majority-based coarse grouping merges. Appendix B.4 compares exact-frequency and coarse grouping on YouTube and Reuters, where their partitions differ.

We additionally evaluated a binary neighborhood-decoupling baseline, which distinguishes between neighbor and non-neighbor relations across modalities, on the attributed-graph datasets:

DatasetGrocery-SReddit-SPhoto
MetricACCNMIFscoreARIACCNMIFscoreARIACCNMIFscoreARI
Neighborhood binary decoupling46.6947.3632.9338.7679.5781.6876.2776.7455.6954.2052.9728.18
Ours47.3248.8734.5940.1281.0482.8977.9878.3156.3654.2653.1728.37

DNI-MMC obtains higher values on all twelve reported dataset–metric combinations. These results provide additional evidence for the full framework in the low-modality setting.

3. Quantitative evidence for frequency-based reliability (W3)

The empirical motivation is presented in Figure 1(b), which reports frequency-specific edge homophily on all five multimodal datasets: CUB, YouTube, NUS-WIDE, AWA, and Reuters. In this part, we add the Pearson correlations between edge-frequency levels and their corresponding homophily ratios in the following table.

DatasetPearson rp-value
YouTube0.95810.00260
Reuters0.93010.00775

These observations support edge frequency as an empirical prior for average semantic reliability for the motivation of neighborhood decoupling.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.