- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces RPAttention, a content-adaptive attention mechanism that enables global communication through a compact set of learned representatives. Experiments show that RPAttention can significantly improve the performance of pyramid vision transformers, such as Swin, PVT, and PVTv2.
Critical strengths
-
I generally like this idea. The representation-driven token aggregation is interesting and well motivated, providing a meaningful alternative to spatially predefined proxies.
-
The decoupled Gather–Interact–Distribute design is intuitive and technically appealing.
-
Extensive evaluations across multiple backbone families and dense prediction tasks demonstrate consistent improvements.
Critical weaknesses
-
Insufficient comparisons with stronger recent methods. Although several efficient attention baselines are included, comparisons with more recent and competitive efficient approaches, such as LRFormer, would better establish the method's standing. In particular, controlled comparisons with alternative token compression strategies would be valuable.
-
Limited evaluation at larger model scales. Most experiments focus on relatively small or medium-sized backbones (Swin-T, PVT-T/S/M). Evaluations on larger vision backbone models, such as PVT-L or Swin-B can strengthen the scalability claims.
-
Efficiency trade-offs remain unclear. The proposed method is not consistently more efficient at standard resolutions. More comprehensive accuracy–latency–memory comparisons under matched settings would strengthen the practical significance.
-
The proposed content-adaptive representative mechanism is interesting, but its applicability beyond conventional vision backbones remains not explored. Evaluating RPAttention on larger-scale architectures (e.g., PVT-L) and, more importantly, modern vision-language models (e.g. LISA) would substantially strengthen the paper. Demonstrating favorable accuracy–efficiency trade-offs in high-resolution multimodal understanding would significantly increase the practical relevance and impact of this work.
Questions for authors
Pls refer to the weakness. The paper quality is good. However, stronger comparisons with recent methods and evaluations on larger backbones would help establish the generality and competitiveness of RPAttention. Exploring its applicability to modern VLMs/MLLMs would further strengthen the significance of this work.
I am generally positive about this paper and would appreciate further evidence or clarification during the rebuttal.
6 comments
Evaluation on Larger Backbones
Thank you for this suggestion. We agree that evaluating RPAttention on larger vision backbones would further assess its scalability and effectiveness.
During the rebuttal period, we conducted an additional ImageNet-1K classification experiment on PVT-L and added the result and related comparisons to Table D.1 of the revised paper:
| Method | Params ↓ | FLOPs ↓ | Top-1 (%) ↑ |
|---|---|---|---|
| PVT-L | 61.4M | 9.8G | 81.7 |
| FLatten-PVT-L | 50.6M | 10.4G | 83.4 |
| InLine-PVT-L | 50.2M | 10.2G | 83.6 |
| Agent-PVT-L | 48.7M | 10.4G | 83.7 |
| RPAttention-PVT-L | 48.2M | 11.1G | 83.6 |
RPAttention improves PVT-L from 81.7% to 83.6% Top-1 accuracy, a gain of 1.9 percentage points, and achieves comparable accuracy to the listed efficient attention methods. This additional result shows that RPAttention remains effective on a larger PVT backbone.
Given the limited rebuttal period, we have focused on adding the PVT-L classification experiment. We agree that evaluations on more large-scale backbones and tasks are valuable. We plan to conduct more comprehensive experiments in future work and update the discussion in the main text accordingly.
Accuracy and Practical Efficiency Trade-offs
We agree that a more comprehensive comparison of accuracy, latency, and memory would help readers assess the practical efficiency of RPAttention. We have therefore added these comparisons in Figures C.2–C.4 and Table C.3 of the revised paper.
At the standard classification resolution, RPAttention is not consistently faster than the original backbone. For example, with inputs, PVT-T has a latency of 5.96 ms, compared with 8.52 ms for RPAttention-PVT-T. At the same time, RPAttention improves ImageNet Top-1 accuracy from 75.1% to 79.2% and reduces peak memory from 93.0 MiB to 83.8 MiB. Compared with Agent-PVT-T, RPAttention-PVT-T improves accuracy from 78.4% to 79.2% with similar latency and memory usage.
At higher resolutions, we further compare execution costs alongside COCO and ADE20K task metrics. The following table presents the PVT-M results:
| Method | COCO box AP ↑ | Latency (ms) ↓ | Memory (MiB) ↓ | ADE20K mIoU ↑ | Latency (ms) ↓ | Memory (MiB) ↓ |
|---|---|---|---|---|---|---|
| PVT-M | 42.0 | 33.39 | 716.2 | 42.91 | 34.96 | 734.5 |
| Agent-PVT-M, uncached | 45.9 | 294.62 | 819.3 | 44.30 | 303.32 | 839.2 |
| Agent-PVT-M, cached | 45.9 | 47.54 | 3229.7 | 44.30 | 48.47 | 3319.7 |
| InLine-PVT-M | 44.0 | 25.44 | 436.9 | 44.59 | 25.23 | 441.2 |
| RPAttention-PVT-M | 45.9 | 26.36 | 427.7 | 45.16 | 27.57 | 432.5 |
All latency and peak memory measurements use an RTX 3090 with FP16 and batch size 1. The high-resolution costs measure the forward pass of classification models adapted to and inputs, respectively; they do not include detection or segmentation heads. For cached Agent Attention, cache construction is excluded from latency, while cached tensors are included in memory.
In this comparison, RPAttention matches Agent Attention’s detection box AP and achieves higher segmentation mIoU, with lower measured latency and memory in both cached and uncached settings. Compared with InLine, RPAttention achieves higher task metrics and similar memory usage, with slightly higher latency. We hope these results help readers assess the trade-offs between accuracy and practical execution costs across tasks and input resolutions.
Thank you again for your suggestion.
More comparisons with stronger recent methods
Thank you for your suggestion. We agree that comparisons with recent methods help establish the position of RPAttention. Table D.1 compares our method with several recent works on corresponding backbones.
Our study focuses on attention-module replacements, whereas LRFormer designs a complete network around its attention mechanism. A direct comparison of final network performance would make it difficult to separate the effects of the attention mechanism from those of the overall architecture. We therefore prioritize comparisons within the same backbone.
Regarding controlled comparisons with alternative token compression strategies, Table 3 includes a k-means routing variant on PVTv2-B0, which achieves 72.6% Top-1 compared with 73.1% for RPAttention. We also conducted a pooled-query Gather comparison on DeiT-Tiny: keeping Interact, Distribute, and DWC unchanged, this variant achieves 75.2%, compared with 75.4% for RPAttention.
We appreciate your recommendation of LRFormer. Its use of a fixed-size, low-resolution representation for global context modeling, combined with local operations for preserving details, shares a motivation with our design. The methods differ in construction and distribution: LRFormer uses spatial pooling and interpolation, while RPAttention constructs representatives through content-dependent weighted aggregation and lets spatial queries read their updated content.
We have included LRFormer in the Related Work section and discussed these similarities and differences in the revision, with the additions highlighted in blue. Thank you again for your suggestion and positive assessment of our work.