acceptodds
Under review as a conference paper at ICLR 2027

Receiver-Query Diversity in Graph Transformers: Frozen Interventions Misestimate Adaptive Capacity

Abstract

Full attention is often valued for its receptive field. Here I investigate a different resource: how many distinct receiver-specific summaries a model can construct from that shared context. Whereas global pooling broadcasts a single uniform representation to all nodes, full attention can tailor a report to each node. I formalize the spectrum between these extremes by parameterizing attention through discrete receiver-query modes. Theoretically, a random-access lower bound and matching construction establish the necessity of query diversity, while controlled graph experiments demonstrate that the demand diversity of receiver tasks, rather than context size alone, governs the required number of modes. Empirically, restricting query diversity within standard Graph Transformers significantly degrades dense node-level segmentation (COCO-SP, PascalVOC-SP), confirming that query diversity represents a primary capacity bottleneck. Crucially, I show that standard frozen ablations, commonly used to assess architectural components, fundamentally misestimate the capacity recoverable through end-to-end adaptation. Across multiple benchmarks, training with constrained queries recovers substantial performance over post-hoc frozen interventions, yet this recovery gap non-monotonically reverses at higher capacities. The findings establish that receiver-query diversity is a genuine capacity axis for Graph Transformers, while demonstrating that frozen sensitivity, adaptive capacity, and routing trainability are distinct properties that standard ablation practices routinely conflate.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph Transformers, Attention Mechanisms, Expressive Capacity, Query Diversity

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_n2CROpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies receiver-query diversity in Graph Transformers: beyond global access, it asks how many distinct global summaries can be tailored to different nodes. The authors introduce (M) query modes between a single shared query and full receiver-specific attention, and provide theoretical and synthetic evidence that limited query diversity can bottleneck tasks requiring different receivers to retrieve different information. On COCO-SP and PascalVOC-SP, sharing one query substantially reduces node-segmentation performance. The paper also shows that frozen query-compression ablations can underestimate the performance achievable after retraining under the same constraint. However, the main positive evidence is limited to two related node-level tasks, and the graph-level Peptides-func experiment does not establish a consistent advantage for greater query diversity.

Critical strengths
  1. The paper introduces a clear and insightful perspective by separating global access from the ability to produce distinct global summaries for different receiver nodes. The notion of receiver-query diversity is intuitive and supported by both theory and controlled experiments.

  2. The paper makes a valuable methodological distinction between frozen post-hoc compression and training under the same capacity constraint. The paired experiments on PascalVOC-SP and Peptides-func show that these settings can lead to substantially different conclusions.

  3. The experimental reporting is transparent and appropriately cautious. The paper retains the negative result on Peptides-func and explicitly avoids claiming a universal capacity law or a practical speedup.

Critical weaknesses
  1. The evidence for generalization is limited. The main positive results come from two closely related superpixel node-segmentation datasets, while the only graph-level task, Peptides-func, does not support the preregistered prediction that greater query diversity improves trained performance. The evaluation does not cover many common graph-learning settings, such as multi-dataset molecular property prediction, link prediction, standard node classification, recommendation graphs, knowledge graphs, traffic or spatiotemporal graphs, graph generation, heterogeneous graphs, or large-scale graphs.

  2. The paper does not provide a practical way to choose the central hyperparameter (M). It evaluates only a few discrete budgets and offers neither a transferable selection rule nor an automatic budget-allocation mechanism.

  3. Practical efficiency is not demonstrated. Although the prototype reduces peak memory by about 24% at (M=16), it is approximately 16% slower than full attention. The work therefore does not establish a full path from a smaller (M) to retained accuracy and real-world efficiency gains.

Questions for authors

See Weakness.

Reviewer_hyqeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates receiver-query diversity as a capacity dimension of Graph Transformers, distinct from the ability to access global graph information. The central idea is that global attention provides not only access to distant nodes but also the ability for different receiving nodes to construct different summaries of the same global context. The authors study this using a query-mode budget MM, ranging from a single shared query to fully receiver-specific attention.

Critical strengths
  1. The paper investigates an interesting distinction between global information access and receiver-specific information aggregation in Graph Transformers.
  2. The comparison between frozen interventions and models trained under the same constraints is useful, and the authors report negative results and failed hypotheses.
Critical weaknesses
  1. The presentation is unnecessarily complicated, with many hypotheses, counter-hypotheses, statistical gates, and auxiliary experiments. The writing and figures do not clearly communicate the main scientific insights. Much of the paper focuses on defending or refining individual experimental claims rather than developing a coherent understanding of the underlying problem.
  2. The main capacity claim is supported by only two closely related superpixel segmentation datasets (COCO-SP and PascalVOC-SP), while the prediction fails on Peptides-func. The complete intermediate-budget training study is conducted only on PascalVOC-SP, using a single Graph Transformer architecture.
  3. The paper varies the number of query modes without investigating interactions with embedding dimension, number of attention heads, or network depth. It is unclear whether the observed limitations persist when the representational capacity of individual summaries increases.
  4. The paper does not convincingly explain when receiver-query diversity constitutes a practical bottleneck in real-world graph problems. The synthetic experiments are designed around the proposed limitation, while the real-world evidence remains narrow.
Questions for authors
  1. What characteristics of real-world graph-learning tasks make receiver-specific query diversity a significant bottleneck? Why should this be considered a general problem for Graph Transformers rather than a limitation of the particular GPS configuration and segmentation tasks studied?

  2. Can the authors provide evidence on additional, structurally different datasets or another Graph Transformer architecture? In particular, why is the complete trained-capacity sweep evaluated on only one dataset?

Reviewer_HWd3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper shows how query diversity affects Graph Transformer capacity beyond global access and whether frozen ablations reveal adaptive performance.

Critical strengths

None

Critical weaknesses

W1: Figure 1 shows clear signs of AI generation and contains almost no useful information. W2: The chapter structure of the entire manuscript is chaotic. It is filled with title like "2 WHY THE NUMBER OF SUMMARIES IS A CAPACITY", which is text clearly generated by AI and makes the manuscript difficult to read.

Questions for authors

The AI tools contribute to nearly every stage of the research process. Could the authors clarify which methodological ideas and technical contributions were independently developed by the research team, and how the AI outputs were validated?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.