acceptodds
Under review as a conference paper at ICLR 2027

Representative Attention for Vision Transformers

Abstract

Many efficient attention methods reduce the quadratic cost of global self-attention by using predefined spatial partitions to construct compact proxies or restrict interactions. However, these partitions can separate distant tokens with similar features and group neighboring tokens with dissimilar features. This creates a mismatch between token interactions and the feature structure of visual content. To address this mismatch, we introduce Representative Attention (RPAttention), which organizes global communication through a compact set of input-dependent representatives. A learned projection of input token features determines their contributions to each representative. RPAttention then constructs representatives by GATHERING information from ALL image tokens rather than a predefined spatial subset. The constructed representatives then INTERACT through lightweight self-attention to refine the gathered content. They DISTRIBUTE their refined content to each spatial query using attention weights computed separately from the gathering assignments. With M representatives and N spatial tokens, the complete Gather–Interact–Distribute process requires O(NM+M^2) token interactions. Experiments on image classification, object detection, instance segmentation, and semantic segmentation demonstrate strong performance across multiple backbone families and competitive results against recent efficient attention methods. Runtime measurements further show favorable scaling as input resolution increases.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision Transformers, Efficient Attention, Linear Attention, Token Compression, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QyeZOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces RPAttention, a content-adaptive attention mechanism that enables global communication through a compact set of learned representatives. Experiments show that RPAttention can significantly improve the performance of pyramid vision transformers, such as Swin, PVT, and PVTv2.

Critical strengths
  • I generally like this idea. The representation-driven token aggregation is interesting and well motivated, providing a meaningful alternative to spatially predefined proxies.

  • The decoupled Gather–Interact–Distribute design is intuitive and technically appealing.

  • Extensive evaluations across multiple backbone families and dense prediction tasks demonstrate consistent improvements.

Critical weaknesses
  1. Insufficient comparisons with stronger recent methods. Although several efficient attention baselines are included, comparisons with more recent and competitive efficient approaches, such as LRFormer, would better establish the method's standing. In particular, controlled comparisons with alternative token compression strategies would be valuable.

  2. Limited evaluation at larger model scales. Most experiments focus on relatively small or medium-sized backbones (Swin-T, PVT-T/S/M). Evaluations on larger vision backbone models, such as PVT-L or Swin-B can strengthen the scalability claims.

  3. Efficiency trade-offs remain unclear. The proposed method is not consistently more efficient at standard resolutions. More comprehensive accuracy–latency–memory comparisons under matched settings would strengthen the practical significance.

  4. The proposed content-adaptive representative mechanism is interesting, but its applicability beyond conventional vision backbones remains not explored. Evaluating RPAttention on larger-scale architectures (e.g., PVT-L) and, more importantly, modern vision-language models (e.g. LISA) would substantially strengthen the paper. Demonstrating favorable accuracy–efficiency trade-offs in high-resolution multimodal understanding would significantly increase the practical relevance and impact of this work.

Questions for authors

Pls refer to the weakness. The paper quality is good. However, stronger comparisons with recent methods and evaluations on larger backbones would help establish the generality and competitiveness of RPAttention. Exploring its applicability to modern VLMs/MLLMs would further strengthen the significance of this work.

I am generally positive about this paper and would appreciate further evidence or clarification during the rebuttal.

6 comments
AuthorsOpenReview ↗

More comparisons with stronger recent methods

Thank you for your suggestion. We agree that comparisons with recent methods help establish the position of RPAttention. Table D.1 compares our method with several recent works on corresponding backbones.

Our study focuses on attention-module replacements, whereas LRFormer designs a complete network around its attention mechanism. A direct comparison of final network performance would make it difficult to separate the effects of the attention mechanism from those of the overall architecture. We therefore prioritize comparisons within the same backbone.

Regarding controlled comparisons with alternative token compression strategies, Table 3 includes a k-means routing variant on PVTv2-B0, which achieves 72.6% Top-1 compared with 73.1% for RPAttention. We also conducted a pooled-query Gather comparison on DeiT-Tiny: keeping Interact, Distribute, and DWC unchanged, this variant achieves 75.2%, compared with 75.4% for RPAttention.

We appreciate your recommendation of LRFormer. Its use of a fixed-size, low-resolution representation for global context modeling, combined with local operations for preserving details, shares a motivation with our design. The methods differ in construction and distribution: LRFormer uses spatial pooling and interpolation, while RPAttention constructs representatives through content-dependent weighted aggregation and lets spatial queries read their updated content.

We have included LRFormer in the Related Work section and discussed these similarities and differences in the revision, with the additions highlighted in blue. Thank you again for your suggestion and positive assessment of our work.

AuthorsOpenReview ↗

Evaluation on Larger Backbones

Thank you for this suggestion. We agree that evaluating RPAttention on larger vision backbones would further assess its scalability and effectiveness.

During the rebuttal period, we conducted an additional ImageNet-1K classification experiment on PVT-L and added the result and related comparisons to Table D.1 of the revised paper:

MethodParams ↓FLOPs ↓Top-1 (%) ↑
PVT-L61.4M9.8G81.7
FLatten-PVT-L50.6M10.4G83.4
InLine-PVT-L50.2M10.2G83.6
Agent-PVT-L48.7M10.4G83.7
RPAttention-PVT-L48.2M11.1G83.6

RPAttention improves PVT-L from 81.7% to 83.6% Top-1 accuracy, a gain of 1.9 percentage points, and achieves comparable accuracy to the listed efficient attention methods. This additional result shows that RPAttention remains effective on a larger PVT backbone.

Given the limited rebuttal period, we have focused on adding the PVT-L classification experiment. We agree that evaluations on more large-scale backbones and tasks are valuable. We plan to conduct more comprehensive experiments in future work and update the discussion in the main text accordingly.

AuthorsOpenReview ↗

Accuracy and Practical Efficiency Trade-offs

We agree that a more comprehensive comparison of accuracy, latency, and memory would help readers assess the practical efficiency of RPAttention. We have therefore added these comparisons in Figures C.2–C.4 and Table C.3 of the revised paper.

At the standard classification resolution, RPAttention is not consistently faster than the original backbone. For example, with 224×224224\times224 inputs, PVT-T has a latency of 5.96 ms, compared with 8.52 ms for RPAttention-PVT-T. At the same time, RPAttention improves ImageNet Top-1 accuracy from 75.1% to 79.2% and reduces peak memory from 93.0 MiB to 83.8 MiB. Compared with Agent-PVT-T, RPAttention-PVT-T improves accuracy from 78.4% to 79.2% with similar latency and memory usage.

At higher resolutions, we further compare execution costs alongside COCO and ADE20K task metrics. The following table presents the PVT-M results:

MethodCOCO box AP ↑Latency (ms) ↓Memory (MiB) ↓ADE20K mIoU ↑Latency (ms) ↓Memory (MiB) ↓
PVT-M42.033.39716.242.9134.96734.5
Agent-PVT-M, uncached45.9294.62819.344.30303.32839.2
Agent-PVT-M, cached45.947.543229.744.3048.473319.7
InLine-PVT-M44.025.44436.944.5925.23441.2
RPAttention-PVT-M45.926.36427.745.1627.57432.5

All latency and peak memory measurements use an RTX 3090 with FP16 and batch size 1. The high-resolution costs measure the forward pass of classification models adapted to 800×1280800\times1280 and 512×2048512\times2048 inputs, respectively; they do not include detection or segmentation heads. For cached Agent Attention, cache construction is excluded from latency, while cached tensors are included in memory.

In this comparison, RPAttention matches Agent Attention’s detection box AP and achieves higher segmentation mIoU, with lower measured latency and memory in both cached and uncached settings. Compared with InLine, RPAttention achieves higher task metrics and similar memory usage, with slightly higher latency. We hope these results help readers assess the trade-offs between accuracy and practical execution costs across tasks and input resolutions.

Thank you again for your suggestion.

Reviewer_QAGWOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a drop-in replacement for the attention mechanism in vision backbones, namely RPAttention. Tokens are softly assigned to several learned anchors and averaged into representatives, the representatives do self-attention, and each spatial query reads them back through cross-attention. A parallel depth-wise convolution branch is also adopted. RPAttention is evaluated on DeiT, PVT, PVTv2 and Swin for ImageNet-1K, COCO and ADE20K.

Critical strengths
  1. The research direction is sound to me, and the method is clearly presented. Routing global communication through a small set of learned representatives is a reasonable way to keep global access without full attention.

  2. The evaluation is broad, and RPAttention improves over every original backbone. Fig. 4 also compares against Agent Attention, FLatten and PolaFormer on the same backbones. This supports the claim that RPAttention works as a general replacement.

Critical weaknesses
  1. The novelty over prior work is limited. Compressing global communication through a small set of intermediate tokens has been widely studied. RPAttention changes representative construction and adds representative interaction, but the ablations in Table 3 do not isolate the benefit of removing Agent Attention's pooling-based spatial prior. I am not convinced that these changes constitute a substantial advance over existing Agent-style methods.

  2. The performance gains over existing attention replacements are limited, and they are not cost-matched. RPAttention matches or exceeds the strongest efficient-attention method in each group (Fig. 4, Table D.1), with Top-1 gains of 0.0–0.8 and only 0.1 on Swin-T, Swin-S and PVT-M. It uses more FLOPs in most groups (e.g., 4.6G vs. 3.8G for MobiAtt on PVTv2-B2 at equal accuracy).

Questions for authors
  1. What do you see as the key limitation of Agent-style global bottlenecks, beyond the pooled initialization of agent tokens, and which part of RPAttention addresses it? Concretely, what does the spatial prior introduced by pooling cost in practice?

  2. What happens if only Gather is replaced with Agent-style pooled queries, keeping Interact, Distribute and DWC unchanged? Conversely, what happens if Interact is added to Agent Attention?

  3. The anchors W_g are static parameters shared across heads. Why were static anchors chosen over input-conditioned anchors, such as the pooled queries used by Agent Attention?

7 comments
AuthorsOpenReview ↗

W1 & Q1:Contribution relative to Agent-style methods

Thank you for your questions. We agree that global communication through a small set of intermediate tokens has been extensively studied. Our question is whether, after an image has already been divided into patch tokens, constructing compact representatives for global communication still requires aggregating tokens within predefined spatial regions.

We consider two effects of the spatial prior introduced by pooling.

First, predefined spatial grouping affects representative construction. Agent Attention constructs agent queries through spatial pooling. When a pooling region contains different content, averaging may mix dissimilar features; spatially distant content in different pooling regions cannot be directly aggregated at this step. RPAttention instead uses competitive, feature-dependent aggregation over all spatial positions, followed by interaction among representatives. Figures 3 and E.1 provide corresponding visualizations of query-to-space responses.

In Appendix E.2, for Stage 4 of PVT-S, RPAttention’s representative-map redundancy is 0.532, compared with 0.807 for Agent Attention, indicating less overlap between its representative maps.

Table 3 primarily analyzes the internal components of RPAttention. To further examine representative construction and interaction, we conducted the two controlled comparisons you suggested. Please see our response to Q2 for the results and analysis.

Second, representative construction also involves a capacity–cost trade-off in high-resolution tasks. An increase in image resolution does not necessarily imply a proportional increase in semantic information, so the number of representatives needed for global communication may not need to grow substantially.

In the released PVT configurations, Agent Attention uses (9,16,49,49) agents for classification and (144,256,784,784) agents for detection and segmentation. Our PVT models instead retain (96,64,49,49) representatives for both classification and downstream tasks.

In addition to the runtime results across resolutions in Table C.1, we provide efficiency comparisons at the input sizes corresponding to COCO and ADE20K. On PVT-M, RPAttention achieves the same detection box AP and higher segmentation mIoU than Agent Attention with fewer FLOPs:

TaskMethodTask metric ↑FLOPs ↓Latency (ms) ↓Memory (MiB) ↓
COCO detectionPVT-M42.0392G33.39716.2
Agent-PVT-M, uncached45.9400G294.62819.3
Agent-PVT-M, cached45.9400G47.543229.7
RPAttention-PVT-M45.9322G26.36427.7
ADE20K segmentationPVT-M42.91315G34.96734.5
Agent-PVT-M, uncached44.30321G303.32839.2
Agent-PVT-M, cached44.30321G48.473319.7
RPAttention-PVT-M45.16241G27.57432.5

The task metrics are box AP for COCO and mIoU for ADE20K. FLOPs correspond to the detection and segmentation models; latency and memory are measured using adapted PVT classification models at the corresponding input sizes on an RTX 3090, with FP16 and batch size 1. The cached rows use fixed-input-shape bias caching: cache construction is excluded from latency, while the cache is included in memory. The cached and uncached rows use the same reported downstream task metrics. Full comparisons are provided in Appendix C.5–C.6, Figures C.2–C.4, and Table C.3 of the revision.

Our contribution therefore focuses on preserving useful content in a compact global representation through feature-driven representative construction and interaction. We have clarified the effect of pooling on agent-query construction in Related Work, Spatial Proxy Construction, and added high-resolution efficiency comparisons to Appendix C.6. Thank you again for your questions.

W2: Evaluating accuracy gains alongside computational cost

Thank you for raising this issue. We agree that, for classification at standard resolution, some RPAttention configurations use more FLOPs than existing attention methods.

Our design aims to support global communication in both classification and high-resolution tasks. As input resolution increases, the number of spatial tokens grows, while the semantic information in an image may not increase proportionally. RPAttention therefore retains the same number of representatives when extending from classification to high-resolution tasks, using feature-based aggregation to maintain a compact global representation.

As shown in our response to W1 and Q1, at input sizes corresponding to COCO and ADE20K, RPAttention-PVT-M has lower latency and memory usage than Agent-PVT-M, while achieving the same detection box AP and higher segmentation mIoU. Latency and memory are measured using PVT classification models adapted to the corresponding input sizes. More comprehensive accuracy–latency–memory comparisons are provided in Appendix C.5–C.6, Figures C.2–C.4, and Table C.3 of the revision.

Regarding MobiAtt, although both methods achieve 82.6% Top-1 on PVTv2-B2, RPAttention also shows accuracy gains in other commonly reported configurations. On DeiT-T and DeiT-S, MobiAtt achieves 73.3% and 80.0%, respectively, while RPAttention achieves 75.4% and 80.8%.

Thank you again for your suggestion. We have added comparisons of accuracy and practical runtime costs at different resolutions to the revised manuscript as a reference for future research and applications.

Q2: Ablating Representative Construction and Interaction

Thank you for suggesting these two insightful comparisons. Following your suggestion, we implemented both variants and trained them on DeiT-Tiny under the same training settings:

MethodImageNet Top-1 ↑
Agent Attention74.9%
Agent Attention + Interact75.1%
RPAttention with pooled-query Gather75.2%
RPAttention75.4%

First, replacing Gather with pooled-query Gather while retaining Interact, Distribute, and DWC reduces accuracy from 75.4% to 75.2%. This comparison supports the value of our representative construction. Although pooled queries can subsequently access all tokens through attention, their retrieval directions are initially determined by features averaged within predefined spatial regions. When a region contains different types of content, this averaging may affect the selectivity of subsequent aggregation. RPAttention instead uses competitive assignment based on token features, allowing content relationships to guide representative construction.

Gather and Interact serve different purposes: Gather determines which content is aggregated into each representative, whereas Interact models relationships among the resulting representatives. Interact can therefore enrich the aggregated representations, but it does not reassign the original spatial tokens. This helps explain why the design of Gather still matters when both variants include Interact.

Second, adding Interact to Agent Attention improves accuracy from 74.9% to 75.1%. This result shows that explicit interaction among agents also provides a benefit in this setting. Before information is distributed back to spatial tokens, each agent can update its content using information from other agents, making their relationships available to subsequent spatial queries. This is consistent with our motivation for performing content interaction within a compact set of representatives.

Together, these comparisons support the two design choices in RPAttention: organizing representatives through feature-driven aggregation and then updating their content through explicit interaction. Their combination achieves the highest accuracy in this experiment. Thank you again for suggesting these comparisons, which help clarify the respective roles of representative construction and interaction.

AuthorsOpenReview ↗

Q3: Why choose static anchors shared across heads?

Thank you for the question. We chose this design to learn a task-dependent rule for grouping tokens, while allowing the current input to determine the content of each representative.

Specifically, WgW_g is trained jointly with the key projection to learn which features should be aggregated into the same representative. For each image, tokens are competitively assigned to representatives based on their current key features, and representatives are then constructed by aggregating across all spatial locations. Thus, WgW_g provides a grouping rule shared across samples, while the spatial regions and content associated with each representative depend on the current image.

Agent Attention’s pooled queries are also input-adaptive, but their construction first averages features within predefined spatial regions. We use learned grouping directions so that representative construction can operate directly on token features without this initial spatial aggregation. This aligns with our motivation to aggregate related content at distant locations while distinguishing adjacent but unrelated content.

Regarding sharing across heads, although WgW_g is shared, each head has a different key projection Wk(h)W_k^{(h)}. The effective grouping directions in the input feature space are therefore Wk(h)WgW_k^{(h)}W_g, which can differ across heads. Sharing WgW_g consequently does not require identical assignments across heads, while reducing the anchor parameter count from hdMhdM for independent anchors to dMdM.

This design retains input adaptivity in token assignments, representative content, and subsequent representative interactions, while implementing global aggregation with a compact parameterization. Thank you again for the opportunity to clarify this design choice.

Reviewer_DmVfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Representative Attention, which replaces standard self-attention in vision transformers. Instead of computing pairwise interactions among all tokens, it uses a three-step process. It first gathers tokens into a small set of learned representatives, then lets these representatives interact with each other, and finally distributes the information back to all tokens. A depth-wise convolution is added to keep local details. The method is tested on classification, detection, and segmentation tasks using several different backbone networks.

Critical strengths
  • The paper addresses the very important problem of the quadratic computational cost in standard self-attention. The goal of reducing this cost while keeping a global receptive field is highly relevant for practical vision models.

  • The proposed three-stage pipeline is well-designed and easy to understand. Using learned feature similarities to group tokens is more flexible than fixed spatial pooling methods, and separating the gathering map from the distributing map is a smart design choice.

  • The authors test their method across many different backbone families, including DeiT, PVT, and Swin. The results show consistent improvements in classification accuracy and dense prediction tasks, which proves the method is broadly applicable.

  • The efficiency analysis is very honest. The authors report that the model is actually slower at low resolutions like 224x224 but becomes much faster at higher resolutions like 1024x1024, showing a clear crossover point instead of just claiming a universal speedup.

Critical weaknesses
  • The core idea has strong similarities with existing efficient attention methods like Agent Attention and FLatten. The paper does not clearly explain what fails if we just use simpler pooling methods, making the main contribution feel somewhat incremental.

  • Several design choices are not well explained or tested. For example, it is unclear why layer normalization is applied only to the representative keys and values, and why the latent queries and keys are derived only from the representative values.

  • The main classification results are only run once. Modest accuracy gains, such as the 0.6 point gain on PVTv2-B2 or the 0.8 point gain on Swin-S, might just be random noise. Multiple random seeds are needed to prove these gains are real.

  • The comparison with other efficient attention methods is only shown as an accuracy versus FLOPs curve. It is hard to see the direct trade-off because the parameter counts are not shown on the same plot, and the models are not compared under strictly matched computational budgets.

Questions for authors
  • Can you provide the standard deviations across multiple random seeds for the main classification table to prove the smaller gains are significant?

  • Why are the layer normalization and the specific projection choices designed this way? Can you show the results if these choices are removed or changed?

  • How does this method compare directly against Agent Attention when using the exact same computational budget and parameter count?

  • When training the model, does the assignment of tokens to representatives collapse to just a few representatives? If so, how do you prevent this?

4 comments
AuthorsOpenReview ↗

W1: Contribution relative to Agent Attention and FLatten, and the motivation beyond pooling

1.1 Thank you for raising this important point. We agree that the paper should more clearly explain how RPAttention differs from existing efficient attention methods and why we use content-dependent representatives rather than simple pooling.

After an image has been divided into patch tokens, pooling further averages token features within predefined spatial regions. When a region crosses object boundaries or contains different content, the resulting representative may mix dissimilar features. RPAttention instead uses competitive assignments based on token features, allowing spatially separated but similar tokens to contribute to the same representative, while adjacent tokens can be assigned to different representatives. These representatives then interact through self-attention and pass the updated information back to spatial tokens.

In the examples shown in Figure 3 and Figure E.1, Agent Attention’s responses are more localized, whereas RPAttention also responds to some spatially separated, related regions. To further quantify these observations, Appendix E.2 compares foreground responses and representative-map redundancy on validation images with localization annotations. For convenience, we reproduce the Stage 4 results below:

MetricAgent AttentionRPAttention
Foreground concentration ↑0.9471.039
Pointing accuracy ↑0.4910.529
Representative-map redundancy ↓0.8070.532

RPAttention achieves higher foreground concentration and pointing accuracy, together with lower representative-map redundancy. These results indicate more focused and diverse representative responses, supporting our motivation for constructing content-dependent representatives.

1.2 FLatten uses focused linear attention, whereas RPAttention performs global communication through Gather–Interact–Distribute. We adopt the simple and effective DWC design from FLatten, which has been widely used, to supplement local information. Even without DWC, RPAttention still achieves a substantial performance gain, as shown in Table 3 and Table D.1:

MethodParamsFLOPsImageNet Top-1
Original PVTv2-B03.7M0.50G70.5
+ FLatten3.6M0.60G71.1
+ RPAttention, w/o DWC3.24M0.71G72.7
+ RPAttention, w/ DWC3.25M0.72G73.1

Without DWC, RPAttention achieves 72.7% Top-1, improving upon the original PVTv2-B0 by 2.2 percentage points and exceeding FLatten’s 71.1% in this comparison. Adding DWC further improves accuracy by 0.4 percentage points, showing that it provides a useful local complement to our global communication mechanism.

Thank you again for helping us identify where our explanation could be clearer. Following your suggestion, we will clarify the motivation for content-dependent representatives and the mechanistic differences between RPAttention, Agent Attention, and FLatten in the revised paper, and present the comparisons and ablations discussed above.

AuthorsOpenReview ↗

W2 & Q2: Motivation for normalization and projection choices

Thank you for pointing out that these two design choices needed clearer explanations.

The input tokens are already normalized by LayerNorm before entering the attention module. After linear projection and Gather aggregation, the feature distributions of the representative keys and values may change, so we normalize them separately after aggregation. The representative keys are used by spatial queries to address the representatives, while the representative values are used for interaction among representatives and subsequent information transfer. The two LayerNorm operations adjust the scales of these aggregated representations.

In Interact, we treat the representative values as the content states and derive queries, keys, and values from them. This allows the representatives to perform self-attention based on their aggregated content. The representative keys are used in the Distribute stage to help spatial queries retrieve the updated representative content.

To investigate the projection choice, we also tried generating the Interact queries and keys from the representative keys, while continuing to use the representative values to convey content. The results on PVT-Tiny are shown below:

Configuration on PVT-TinyImageNet Top-1
Original RPAttention79.2
Interact queries and keys generated from representative keys78.8

Using representative keys to define Interact reduces Top-1 accuracy by 0.4 percentage points. This result is consistent with our design intuition: values carry the content aggregated by each representative and are used for content interaction, while keys provide the routing information that helps spatial queries retrieve the relevant representatives.

Thank you again for your suggestion. We have added an explanation of the motivations behind the normalization and projection choices in the revised paper.

W3 & Q1 Stability of the classification improvements across random seeds

Thank you for raising this important point. We agree that experiments with multiple random seeds would help assess the stability of the performance gains more clearly. However, due to time constraints, we have not yet been able to complete these experiments.

For the results on larger models, we also believe the gains should be considered in the context of their original performance levels. Swin-S + RPAttention achieves 83.8% Top-1, exceeding the reported 83.5% of the larger Swin-B, which illustrates the practical value of this improvement.

In addition, during the rebuttal period, we conducted a classification experiment on the larger PVT-L backbone and added the result to Table D.1 of the revised paper. RPAttention improves PVT-L’s Top-1 accuracy from 81.7% to 83.6%, showing that the method also performs well on this larger backbone.

Thank you again for your suggestion. We hope to conduct repeated training runs in future work to further assess the stability of these gains.

W4 & Q3 Direct comparison with other efficient attention methods

Thank you for raising this point. We agree that an accuracy–FLOPs curve alone does not fully capture the parameter counts and practical execution costs of different methods. We have therefore added accuracy–latency–memory comparisons at the corresponding resolutions for ImageNet classification, COCO detection, and ADE20K segmentation in Appendices C.5–C.6, Figures C.2–C.4, and Table C.3 of the revised paper. The parameter counts and FLOPs of the classification models are reported in Table D.1.

The table below presents selected results for the original PVT backbone, Agent Attention, and RPAttention:

TaskModelParams ↓FLOPs ↓Task metric ↑Latency (ms) ↓Memory (MiB) ↓
ImageNet classificationPVT-T13.2M1.9G75.15.9693.0
Agent-PVT-T11.6M2.0G78.48.6982.3
RPAttention-PVT-T11.5M2.2G79.28.5283.8
COCO detectionPVT-M—392G42.033.39716.2
Agent-PVT-M, uncached56M400G45.9294.62819.3
Agent-PVT-M, cached56M400G45.947.543229.7
RPAttention-PVT-M—322G45.926.36427.7
ADE20K segmentationPVT-M48M315G42.9134.96734.5
Agent-PVT-M, uncached40M321G44.30303.32839.2
Agent-PVT-M, cached40M321G44.3048.473319.7
RPAttention-PVT-M39M241G45.1627.57432.5

The task metrics for ImageNet, COCO, and ADE20K are Top-1 accuracy, box AP, and mIoU, respectively.

At the classification resolution, RPAttention and Agent Attention have similar parameter counts and computational costs, while RPAttention achieves higher accuracy. At the tested high resolutions, RPAttention’s latency and memory advantages become more apparent. For example, at the input resolution corresponding to ADE20K, RPAttention-PVT-M reduces latency by approximately 90.9% and 43.1%, and peak memory by 48.5% and 87.0%, compared with uncached and cached Agent-PVT-M, respectively.

These comparisons use backbones of the same scale. Thank you again for your suggestion, please refer to the revised paper for the complete results and measurement settings.

AuthorsOpenReview ↗

Q4: Representative utilization and assignment collapse

Thank you for raising this important point. Your concern is well founded, as we do not explicitly require all representatives to be used equally.

To investigate this, we analyzed representative utilization in a trained PVT-Tiny model using 5,000 ImageNet validation images. For each image and attention head, we first computed the average assignment weight pmp_m for each representative. We then calculated the effective number of representatives from the entropy of the weight distribution: Neff=exp⁡(−∑mpmlog⁡pm)N_{\mathrm{eff}}=\exp(-\sum_m p_m\log p_m). A value closer to the total number of representatives indicates a more broadly distributed set of weights.

The final block contains 49 representatives. The results are as follows:

StageTotal representativesMean effective representatives
Gather4920.24
Distribute4937.02

These results show that, although utilization during Gather is not perfectly balanced, assignments do not concentrate on only a few representatives. During Distribute, spatial queries read from a broader set of representatives.

We do not use an additional load-balancing loss. RPAttention produces soft assignments based on token features and distinct learnable directions, allowing different representatives to aggregate content from the entire image. Each representative’s aggregation weights are then normalized by its total received weight, yielding a weighted average of the corresponding token features. Interact further enables representatives to share information through self-attention, after which spatial queries retrieve the updated representatives according to their own content. Together, these designs allow multiple representatives to participate in global information transfer without forcing them to receive equal assignment weights.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Further Discussion

Dear Reviewers,

Thank you again for your thoughtful feedback, which has been valuable in improving our work. We have submitted a revised version of the paper with additional experiments, analyses, and clarifications in response to your comments.

We welcome further discussion and would be happy to address any remaining questions or concerns. We look forward to hearing your thoughts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.