acceptodds
Under review as a conference paper at ICLR 2027

Cluster Attention for Graph Deep Learning

Abstract

Message Passing Neural Networks have recently become the most popular approach to graph machine learning tasks; however, their receptive field is limited by the number of message passing layers. To increase the receptive field, Graph Transformers with global attention have been proposed; however, global attention does not take into account the graph topology and thus lacks graph-structure-based inductive biases, which are typically very important for graph machine learning tasks. In this work, we propose an alternative approach: cluster attention (CLATT). We divide graph nodes into clusters with off-the-shelf graph community detection algorithms and let each node attend to all other nodes in each cluster. CLATT provides large receptive fields while still having strong graph-structure-based inductive biases. We show that augmenting Message Passing Neural Networks or Graph Transformers with CLATT significantly improves their performance on a wide range of graph datasets including datasets from the recently introduced GraphLand benchmark representing real-world applications of graph machine learning.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
graphs, graph neural networks, GNNs, graph transformers, attention, graph clustering, community detection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vbGROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Cluster Attention (CLATT), a graph learning mechanism that allows nodes within the same cluster to exchange information through attention. By using graph clustering as a preprocessing step, CLATT aims to capture longer-range dependencies while preserving structural information from the graph. The method can be integrated into both message-passing neural networks and Graph Transformers. The authors consider four clustering algorithms and select suitable clusterings for different datasets. Experiments on 12 graph datasets show that adding CLATT generally improves the performance of different graph learning models.

Critical strengths
  1. The paper addresses an important problem in graph learning: how to capture longer-range dependencies while retaining useful structural information. Using graph clusters to define the attention scope provides a reasonable middle ground between local message passing and global attention.
  2. CLATT can be integrated into different graph learning architectures without replacing their original message-passing or attention mechanisms. The experiments include GCN, GraphSAGE, LGT, and two Global Graph Transformer variants, showing that the method can be applied to different types of models.
  3. The experimental evaluation covers 12 datasets with different graph structures and learning tasks, including both homophilous and heterophilous graphs. This provides relatively broad evidence for the usefulness of cluster attention.
  4. The paper considers several graph clustering algorithms and examines the attention distances learned by different models. These analyses help explain how cluster attention differs from local and global attention in terms of the nodes that can interact.
Critical weaknesses

1.The core idea of CLATT is to partition graph nodes using existing graph clustering algorithms, perform standard multi-head self-attention within each cluster, and then combine the outputs with the original message passing or global attention. Although this design has a clear intuitive motivation, its main components are based on existing techniques, including community detection, grouped self-attention, and multi-branch feature fusion. The authors should explain in more detail how the proposed design differs from existing sparse attention, grouped attention, and hierarchical graph modeling methods, and clarify where its advantages come from and which components contribute to them. 2.The paper claims that modern graph clustering algorithms run much faster than GNN training, making the clustering preprocessing time negligible. However, CLATT may use multiple clustering algorithms simultaneously, and the computational costs of different clustering algorithms can vary significantly on large-scale graphs. Does this claim still hold in such cases? The authors should provide a more detailed explanation or additional experimental evidence to support it. 3.In Table 2, the paper uses underlining to indicate statistically significant performance differences, but does not sufficiently explain the statistical test used, the significance level, or how the test was conducted. Reporting only the mean and standard deviation is also insufficient to reproduce the significance testing results. The authors should clarify these details to improve the transparency and reproducibility of the experiments. 4.The experiments mainly evaluate CLATT by comparing the original GCN, GraphSAGE, LGT, and GGT models with their CLATT-augmented versions. However, the augmented models not only change the range of information exchange between nodes but also introduce additional query, key, and value projections, increasing the number of model parameters and the computational cost. Therefore, it is difficult to determine whether the performance improvements come from the increased model capacity or from the cluster attention mechanism itself.

Questions for authors

1.The paper considers four graph clustering algorithms, but the main experiments do not include a random grouping baseline with matched cluster sizes. I suggest adding random grouping and simple structure-based grouping baselines with matched cluster sizes to better evaluate the independent contribution of graph community structure to the final performance. 2.The paper treats the selection of clustering algorithms as a dataset-level hyperparameter, but in practice, it only uses LGT to select the clustering algorithms and assumes that the selected clusterings transfer well across different models, without providing experiments to verify this assumption. Since different models have different receptive fields, attention mechanisms, and abilities to capture graph structure, the clustering combination that works best for LGT may not necessarily be the best choice for other architectures. I suggest selecting clustering algorithms separately for different models on at least a few representative datasets and comparing model-specific selection with shared selection to verify whether this assumption is reasonable.

5 comments
AuthorsOpenReview ↗

Response to Reviewer vbGR (Part 1)

Dear reviewer, thank you for your positive assessment and valuable feedback. We address your concerns below.

1.The core idea of CLATT is to partition graph nodes using existing graph clustering algorithms, perform standard multi-head self-attention within each cluster, and then combine the outputs with the original message passing or global attention. Although this design has a clear intuitive motivation, its main components are based on existing techniques, including community detection, grouped self-attention, and multi-branch feature fusion. The authors should explain in more detail how the proposed design differs from existing sparse attention, grouped attention, and hierarchical graph modeling methods, and clarify where its advantages come from and which components contribute to them.

Indeed, our method builds on existing techniques, however, we believe the specific way we combine them is novel and very different from prior works (and is empirically very useful, as our experiments show). While there are prior works that use some form of sparse attention in Graph Transformers [1], to our knowledge none of them use graph clustering to specify which node pairs can and cannot attend to each other, which is our key contribution. Other prior works use graph pooling and/or hierarchical modeling to iteratively merge nodes into supernodes, which essentially performs a form of learnable node clustering [2-5]. However, these approaches lose information about individual nodes when merging them, and thus cannot be applied to node-level tasks (which are the focus of our work), and are rather applied to graph-level tasks like molecule classification. Further, this works allow interactions between supernodes (which represent node clusters), which is inter-cluster interaction, while our method allows interactions between nodes within the same cluster, which is intra-cluster interaction. Moreover, these works learn to perform node clustering and the target task simultaneously, which significantly complicates the training pipeline, while our method relies on off-the-shelf node clustering methods as a cheap preprocessing step and thus not only simplifies training, but also enables the reuse of decades of prior research on classic community detection algorithms. Finally, some works use attention-based models to solve the task of node clustering [6, 7]. This is very different from our work, which uses node clustering as a preprocessing step to restrict attention and solve standard node-property prediction tasks. Overall, while there are some prior works bridging sparse attention and Graph Transformers or graph clustering and GNNs, none of them use classic graph clustering to restrict attention between nodes in the graph, which is the key method of our work and is our novel approach. We will extend the Related Work section to discuss these papers and how our method differs from them.

[1] NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classification (NeurIPS 2022)

[2] Hierarchical Transformer for Scalable Graph Learning (arxiv preprint, 2023)

[3] Hierarchical Graph Pooling with Structure Learning (arxiv preprint, 2023)

[4] Hierarchical message-passing graph neural networks (DMKD 2023)

[5] Hierarchical Attention-based Graph Neural Network with Relevance-driven Pruning (arxiv preprint, 2026)

[6] Transforming Graphs for Enhanced Attribute Clustering: An Innovative Graph Transformer-Based Method (arxiv preprint, 2023)

[7] Attention-based Graph Clustering Network with Dual Information Interaction (Knowledge-Based Systems, 2025)

AuthorsOpenReview ↗

Response to Reviewer vbGR (Part 2)

2.The paper claims that modern graph clustering algorithms run much faster than GNN training, making the clustering preprocessing time negligible. However, CLATT may use multiple clustering algorithms simultaneously, and the computational costs of different clustering algorithms can vary significantly on large-scale graphs. Does this claim still hold in such cases? The authors should provide a more detailed explanation or additional experimental evidence to support it.

Thank you for this question. Indeed, the speed of different clustering algorithms differs significantly, however, as we discuss in Section 4, for our work we have selected 4 algorithms that run very fast even on large graphs (specifically, they are all almost linear in graph size). Empirically, our largest graph is pokec-regions with 1.5+M nodes and 22+M edges, and each of the considered algorithms takes no more than 2 minutes to cluster this graph (with all 4 algorithms together finishing in less than 5 minutes), which is quite negligible compared to each model training run which takes approximately 30 minutes (and we perform many training runs for hyperparameter selection, while clustering is performed once). We will add the time measurements to the updated version of our paper.

3.In Table 2, the paper uses underlining to indicate statistically significant performance differences, but does not sufficiently explain the statistical test used, the significance level, or how the test was conducted. Reporting only the mean and standard deviation is also insufficient to reproduce the significance testing results. The authors should clarify these details to improve the transparency and reproducibility of the experiments.

Thank you for pointing this out. All experiments were repeated 10 times with different random seeds, the t-test with a significance level of 5% was used. We will clarify this is the updated version of our paper.

4.The experiments mainly evaluate CLATT by comparing the original GCN, GraphSAGE, LGT, and GGT models with their CLATT-augmented versions. However, the augmented models not only change the range of information exchange between nodes but also introduce additional query, key, and value projections, increasing the number of model parameters and the computational cost. Therefore, it is difficult to determine whether the performance improvements come from the increased model capacity or from the cluster attention mechanism itself.

Indeed, our CLATT mechanism introduces additional model parameters. However, for each model (both CLATT and non-CLATT versions) extensive hyperparameter tuning was performed, including sweeps across the number of layers and the hidden dimension, so each model could increase its parameter count if it is useful for the task, but the best versions of all models (by the validation set performance) all turned out to be smaller than the maximum size allowed in our sweeps (and in fact, optimal CLATT versions of models are typically less deep than the corresponding non-CLATT versions, which suggests that the inclusion of longer-range interactions by CLATT allows decreasing the number of message passing layers).

AuthorsOpenReview ↗

Response to Reviewer vbGR (Part 3)

1.The paper considers four graph clustering algorithms, but the main experiments do not include a random grouping baseline with matched cluster sizes. I suggest adding random grouping and simple structure-based grouping baselines with matched cluster sizes to better evaluate the independent contribution of graph community structure to the final performance.

This is indeed a good suggestion. To verify that the cluster structure is what makes CLATT perform well and that simple random groupings are not enough, we conducted experiments with randomized clusters of the same size as you suggest. Further, besides completely randomized clusters, we also tried partially randomized clusters, where for each cluster a certain percentage of nodes is taken from the original cluster, and the remaining nodes are assigned randomly. Due to a large number of experiments, we limit this investigation to our strongest model (LGT) and 5 datasets (please let us know if you would like to see more models or datasets, we can add them later). We provide the experimental results in the table below. No-CLATT stands for our baseline – LGT without CLATT. CLATT-random-X% stands for using CLATT with X% of nodes being assigned to clusters randomly, i.e. CLATT-random-0% uses the original clustering with no randomness, and CLATT-random-100% uses a completely random clustering, while the levels of randomness of 10%, 25%, and 50% lie between these two extremes.

datasetcity-roads-Mlastfm-asiafacebookamazon-ratings-5corehm-categories
No-CLATT58.0583.9792.1751.2669.25
CLATT-random-100%58.0283.9192.2051.1069.31
CLATT-random-50%58.0083.9492.2051.2769.23
CLATT-random-25%58.4084.1292.4852.3569.53
CLATT-random-10%59.7184.4192.6053.0869.92
CLATT-random-0% (orig)60.2684.5592.7153.7070.22

We can see that the model with 100% of randomized cluster assignments performs on the level of the No-CLATT baseline, i.e., the model likely learns to ignore the CLATT mechanism since it provides no useful information when clusters are completely randomized. CLATT with 50% of noise also performs similar to the baseline, suggesting that such a randomness level is still too high to extract useful information. CLATT with 25% of randomness performs better than the baseline, while CLATT with 10% of randomness performs close to the original CLATT, suggesting that the CLATT mechanism can withstand low randomness levels and retain most of its performance benefits. Overall, we can see that the cluster structure is what explains the practical effectiveness of CLATT, as the larger parts of the clusters get randomized, the worse performance CLATT achieves, and CLATT with completely randomized clusters does not outperform the baseline model without CLATT.

Reviewer_q4deOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adds cluster-restricted attention to existing graph learning models. The graph is partitioned using community detection algorithms, and nodes attend to other nodes within their assigned cluster. The authors consider several graph-based clusterings and k-means on learned node representations. They select useful clusterings on the validation set and combine the outputs from the seperate partitions with the backbone representations. Experiments cover 12 node prediction datasets, three message-passing architectures, and two global transformer variants. The main finding is that adding CLATT improves performance in nearly all reported comparisons. The appendix also examines attention distances to illustrate how cluster attention differs from local and global attention.

Critical strengths
  • The strongest part of the paper is the breadth and consistency of the results. Table 2 shows higher mean performance in 55 of the 56 completed comparisons, across datasets with quite different sizes and structural properties. The gains are not confined to the weaker baselines: LGT improves from 46.41 to 49.34 on pokec-regions, for example. This is fairly consistant evidence that the augmentation is useful in the evaluated setting.

  • The method is simple and can be added to several backbones. I also like the decision to consider different kinds of graph partitions, including statistical clustering that can capture disassortative structure. These partitions could expose relationships that ordinary neighborhood aggregation misses. Using strengthened MPNN baselines and testing the full Amazon graph, rather than only its potentially favorable 5-core, are sensible evaluation choices.

Critical weaknesses
  • I am not yet convinced that the paper establishes its contribution relative to closely related work. CoBFormer, “Less is More: on the Over-Globalizing Problem in Graph Transformers” (ICML 2024), already partitions a graph with METIS and applies dense Transformer attention within each cluster. Its intra-cluster attention in Section 4.1, Eq. (9), is closely related to CLATT’s Eqs. (2)–(5). CoBFormer also includes inter-cluster attention and a GCN component, so the complete architectures differ. CLATT’s use of multiple partitioning algorithms and its integration into different backbones may be useful contributions, but the paper needs to explain and evaluate these differences. There is currently no discussion or direct comparision. Table 4 also only identifies the selected clusterings; it does not show whether combining them improves over the best single clustering.

  • There is a substantial gap between the described method and the released implemenation. The code removes entire clusters outside a size interval, normally [4, 512], but this step is not described in the paper. With the supplied partitions, this excludes 88.25% of nodes from the sole selected partition on avazu-ctr and 57.42% on pokec-regions. This matters for understanding both the actual operation and its computational cost. There is also a concrete indexing issue: excluded nodes retain backward indices of (-1, -1), which are used directly to retrieve attention outputs. On questions, the supplied k-means partition contains a 7,301-node cluster that exceeds the configured limit of 2048. All these nodes consequently retrieve the attention output for node 48812 in another cluster, because the final retained cluster has no padding at its last position. This does not implement the stated within-cluster attention. I cannot tell whether this code version was used for the reported results, but that needs to be clarified and the affected results checked.

Questions for authors
  • How does CLATT differ from CoBFormer’s intra-cluster attention beyond the partitioning algorithm and backbone integration? A focused comparison on one or two representative datasets, using the same splits and tuning budget, would help assess the additional contribution.

  • Were the cluster-size filtering and backward-indexing implementations in the released code used to obtain Table 2? Please explain how the size limits were choosen and what excluded nodes should receive as their cluster-attention representation. Please also report node coverage after filtering and verify the questions results after fixing the indexing issue.

  • Does the combination of selected clusterings outperform the best single clustering? Table 4 does not answer this. A small ablation comparing the seperate clusterings, their combination, and random partitions with the same cluster-size distribution would help establish whether the gains come from meaningful graph partitions and their complementary information.

5 comments
AuthorsOpenReview ↗

Response to Reviewer q4de (Part 1)

Dear reviewer, thank you for your valuable feedback. We address your concerns below.

I am not yet convinced that the paper establishes its contribution relative to closely related work. CoBFormer, “Less is More: on the Over-Globalizing Problem in Graph Transformers” (ICML 2024), already partitions a graph with METIS and applies dense Transformer attention within each cluster. Its intra-cluster attention in Section 4.1, Eq. (9), is closely related to CLATT’s Eqs. (2)–(5). CoBFormer also includes inter-cluster attention and a GCN component, so the complete architectures differ. CLATT’s use of multiple partitioning algorithms and its integration into different backbones may be useful contributions, but the paper needs to explain and evaluate these differences. There is currently no discussion or direct comparision.

How does CLATT differ from CoBFormer’s intra-cluster attention beyond the partitioning algorithm and backbone integration? A focused comparison on one or two representative datasets, using the same splits and tuning budget, would help assess the additional contribution.

Thank you for pointing out this work to us. Indeed, the CoBFormer paper is a highly relevant work which we were not previously aware of and which we will add to the discussion in the Related Work section. The intra-cluster attention module of CoBFormer is indeed similar to our CLATT mechanism. However, we believe there are 2 substantial differences between CoBFormer and our work. First, CobFormer is a highly complex and involved architecture which besides intra-cluster attention includes inter-cluster attention (which together with intra-cluster attention can approximate global attention), a local module, and different mechanisms for combining them. Further, a rather complicated collaborative training pipeline is used for training CoBFormer. It is not clear which parts of this architecture and pipeline are responsible for CoBFormer’s success. In contrast, we specifically propose CLATT as a simple plug-and-play module which can be easily combined with any MPNN or Graph Transformer to improve their results. Second, CoBFormer is co-designed specifically with the METIS graph clustering algorithm. In contrast, we emphasize in our work that CLATT can be combined with any graph clustering algorithms (or even several of them simultaneously) and show that clustering algorithm selection can noticeably improve its performance.

Ultimately, we believe that the best way to justify the differences between CLATT and CoBFormer is to directly compare them empirically. Due to limited computational resources, we provide this comparison only for the 7 GraphLand datasets (please let us know if you would like to see results on the remaining datasets, we might be able to add them later). We provide an experimental comparison between CoBFormer, GCN-CLATT (since CoBFormer uses a GCN as its local component), and LGT-CLATT (generally the strongest of our CLATT-augmented models) in the Table below.

datasetcity-roads-Mcity-roads-Lavazu-ctrtolokers-2city-reviewshm-categoriespokec-regions
CoBFormer59.50 ± 0.1353.89 ± 0.2830.78 ± 0.3153.71 ± 0.7577.10 ± 0.0859.78 ± 0.3137.86 ± 1.12
GCN-CLATT59.78 ± 0.2855.15 ± 0.2030.74 ± 0.2654.08 ± 0.6177.79 ± 0.1864.96 ± 0.2448.88 ± 0.44
LGT-CLATT60.26 ± 0.2855.75 ± 0.3431.48 ± 0.4556.95 ± 0.2877.40 ± 0.1670.22 ± 0.2949.34 ± 0.29

We can see that GCN-CLATT outperforms CoBFormer on 6/7 datasets, while LGT-CLATT outperforms CoBFormer on all 7 datasets, suggesting that our CLATT is generally a more preferable method. The differences are particularly large on the hm-categories and pokec-regions datasets.

AuthorsOpenReview ↗

Response to Reviewer q4de (Part 2)

Table 4 also only identifies the selected clusterings; it does not show whether combining them improves over the best single clustering.

Does the combination of selected clusterings outperform the best single clustering? Table 4 does not answer this. A small ablation comparing the seperate clusterings, their combination, and random partitions with the same cluster-size distribution would help establish whether the gains come from meaningful graph partitions and their complementary information.

Thank you for this suggestion. Below we provide a Table with this information. For each dataset for which multiple clusterings have been selected (a total of 6 datasets), we provide results with these multiple clusterings (mult. clust.) and with the best single clustering (best single clust.) for GCN, GraphSAGE, and LGT.

datasetcity-roads-Mcity-roads-Lquestionslastfm-asiafacebookamazon-ratings-full
GCN59.0553.2618.6080.6291.0039.28
GCN-CLATT (best single clust.)59.5054.8019.7283.4691.5839.61
GCN-CLATT (mult. clust.)59.7855.1520.4884.0191.7139.76
GraphSAGE57.5152.4320.4180.9091.2238.73
GraphSAGE-CLATT (best single clust.)60.0255.0320.7883.8991.9138.98
GraphSAGE-CLATT (mult. clust.)60.6755.5920.8984.5492.0939.21
LGT58.0553.3817.8683.9792.1739.03
LGT-CLATT (best single clust.)59.9055.1619.3184.3892.5139.25
LGT-CLATT (mult. clust.)60.2655.7521.2884.5592.7139.34

We can see that using a single clustering is always helpful (note that this clustering was selected based on validation set performance, but the table provides test set performance), however, using multiple clusterings allows further improving performance, and these improvements range from rather small to quite substantial.

AuthorsOpenReview ↗

Response to Reviewer q4de (Part 3)

There is a substantial gap between the described method and the released implemenation. The code removes entire clusters outside a size interval, normally [4, 512], but this step is not described in the paper.

Indeed, we do not use clusters that contain less than 4 or more than 512 nodes. We will explicitly discuss this in the updated version of our paper. We exclude these clusters primarily for efficiency purposes since we create a padded tensor of shape [num_clusters, max_cluster_length, hidden_dim], and including very large clusters or a large number of very small clusters will be inefficient (due to requiring a lot of padding). However, there are also reasons stemming from the expected behaviour of our method to exclude these clusters. We exclude clusters with less than 4 nodes since nodes in such small clusters can easily interact via message passing and do not require cluster attention. We exclude clusters with more than 512 nodes as such large clusters are unlikely to provide useful information about node labels (they cover very large portions of the graph with likely diverse labels). Please note that these limits for cluster sizes were chosen prior to any experiments and were not tuned. The only change we did after computing clusterings (but still prior to training any models) is we increased the maximum cluster size to 2048 for the questions dataset, since we noticed that on this dataset in particular the clustering algorithms mostly produced rather large clusters. We will add this discussion to the updated version of our paper.

There is also a concrete indexing issue: excluded nodes retain backward indices of (-1, -1), which are used directly to retrieve attention outputs. On questions, the supplied k-means partition contains a 7,301-node cluster that exceeds the configured limit of 2048. All these nodes consequently retrieve the attention output for node 48812 in another cluster, because the final retained cluster has no padding at its last position. This does not implement the stated within-cluster attention. I cannot tell whether this code version was used for the reported results, but that needs to be clarified and the affected results checked.

The behaviour of this part of the code is indeed not quite obvious, however, there is no indexing mistake/bug here. Due to clusters being sorted by length and then padded to the maximum length, the index (-1, -1) always corresponds to a padding token in our experiments. Since a mask (constructed according to cluster lengths) is used in attention, the padding tokens never actually take part in attention and remain represented by vectors of zeros. Thus, the output of the attention operation for excluded nodes is a vector of zeros, as expected.

Please let us know if you have any remaining concerns.

Reviewer_q7V5OpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a simple but apparently effective idea: an intermediate solution between the local message passing used in GNNs and the global attention used in Graph Transformers. In particular, the authors propose using the clusters to which nodes belong. The clusters are computed in advance using different algorithms, and multiple partitions can be used. The approach, CLATT, is added to existing GNNs and Graph Transformers and yields improvements in almost all the combinations tested.

Critical strengths

The method is simple and modular, so it does not require a complete change to the architecture. Experiments cover different types of graphs, and the results are very good.

Critical weaknesses

CLATT introduces both additional parameters and the ability to communicate beyond immediate neighbours. To understand whether the cluster structure itself matters, CLATT should be compared with a version using random groups of the same sizes.

For each dataset, the authors train the model with different clusterings and select those that perform best on the validation data. These trials require additional computation, and there is a risk that the selection works particularly well on those data without providing the same benefit on new data.

Questions for authors

The total computational cost of selecting among the different clusterings is unclear to me. Could the authors clarify this?

3 comments
AuthorsOpenReview ↗

Response to Reviewer q7V5 (Part 1)

Dear reviewer, we thank you for your positive feedback. We address your concerns below.

CLATT introduces both additional parameters and the ability to communicate beyond immediate neighbours. To understand whether the cluster structure itself matters, CLATT should be compared with a version using random groups of the same sizes.

This is indeed a good suggestion. To verify that the cluster structure is what makes CLATT perform well, we conducted experiments with randomized clusters of the same size as you suggest. Further, besides completely randomized clusters, we also tried partially randomized clusters, where for each cluster a certain percentage of nodes is taken from the original cluster, and the remaining nodes are assigned randomly. Due to a large number of experiments, we limit this investigation to our strongest model (LGT) and 5 datasets (please let us know if you would like to see more models or datasets, we can add them later). We provide the experimental results in the table below. No-CLATT stands for our baseline – LGT without CLATT. CLATT-random-X% stands for using CLATT with X% of nodes being assigned to clusters randomly, i.e. CLATT-random-0% uses the original clustering with no randomness, and CLATT-random-100% uses a completely random clustering, while the levels of randomness of 10%, 25%, and 50% lie between these two extremes.

datasetcity-roads-Mlastfm-asiafacebookamazon-ratings-5corehm-categories
No-CLATT58.0583.9792.1751.2669.25
CLATT-random-100%58.0283.9192.2051.1069.31
CLATT-random-50%58.0083.9492.2051.2769.23
CLATT-random-25%58.4084.1292.4852.3569.53
CLATT-random-10%59.7184.4192.6053.0869.92
CLATT-random-0% (orig)60.2684.5592.7153.7070.22

We can see that the model with 100% of randomized cluster assignments performs on the level of the No-CLATT baseline, i.e., the model likely learns to ignore the CLATT mechanism since it provides no useful information when clusters are completely randomized. CLATT with 50% of noise also performs similar to the baseline, suggesting that such a randomness level is still too high to extract useful information. CLATT with 25% of randomness performs better than the baseline, while CLATT with 10% of randomness performs close to the original CLATT, suggesting that the CLATT mechanism can withstand low randomness levels and retain most of its performance benefits. Overall, we can see that the cluster structure is what explains the practical effectiveness of CLATT, as the larger parts of the clusters get randomized, the worse performance CLATT achieves, and CLATT with completely randomized clusters does not outperform the baseline model without CLATT.

AuthorsOpenReview ↗

Response to Reviewer q7V5 (Part 2)

For each dataset, the authors train the model with different clusterings and select those that perform best on the validation data. These trials require additional computation, and there is a risk that the selection works particularly well on those data without providing the same benefit on new data.

The total computational cost of selecting among the different clusterings is unclear to me. Could the authors clarify this?

We would like to clarify that in all our experiments using models with and without CLATT, we conducted 30 training runs to choose model hyperparameters (based on the performance on the validation set). In experiments with CLATT, we additionally conducted 4 more training runs to choose the clusterings to use (1 run per 1 potential clustering algorithm, as described in Section 4). Thus, adding CLATT only adds 4 additional training runs to 30 original training runs, which is a rather small cost considering the strong gains in prediction quality. Moreover, as we also describe in Section 4, to further reduce the computational costs being added when using CLATT, these 4 additional training runs were only conducted for one of the considered model (specifically, LGT), and then the clustering algorithms selected for LGT were reused for all other models. Since we experiment with a total of 5 models, this means that we only add 4 training runs to the original 5 * 30 = 150 training runs, which is rather negligible. It is actually quite possible that we could further improve the predictive performance of the other 4 models if we tuned the clustering selection for them rather than simply reusing the selection made for LGT, but we decided to not do it to show that even when transferring the clustering selection between models CLATT still works very well and outperforms the baseline model versions without CLATT.

Please let us know if you have any remaining concerns.

Dear Reviewer,

As the discussion period is ending soon, we would like to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you would like us to consider, please let us know. Thank you for your time and effort in reviewing our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.