- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SCA-DETR, a method for fusing multi-scale features from hierarchical backbones in DETR-based detectors. SCA uses the scale relationship between adjacent stages to define local neighborhoods, then combines attention, gated fusion, and convolutional refinement along top-down and bottom-up paths. On COCO, the evaluated variants improve accuracy over matched RT-DETR and DEIM-D-FINE baselines while reducing latency under the reported inference settings.
Critical strengths
I think making more effective use of multi-scale features remains an important problem for DETR-based detectors. SCA offers a practical fusion module that works with several detector frameworks without changing their decoders or detection heads. The gains in accuracy and measured latency support its practical value.
Critical weaknesses
The diagnostics do not clearly explain why a small, fixed neighborhood is sufficient. Swin-T shows much larger impulse-response drift and lower cross-stage cosine similarity than ResNet-50. However, showing that the drift stays within an architecture-derived bound does not establish that SCA’s local windows cover the information needed for cross-stage fusion. Impulse-response drift does not directly measure cross-stage correspondence, and low cosine similarity may reflect differences in feature representations rather than spatial misalignment. The connection between these observations and the chosen window sizes needs more explanation.
The advantage over deformable attention is unclear. Deformable attention already gathers sparse cross-scale features around reference points, and Lite DETR’s key-aware variant also uses sampled keys to compute attention weights. It is therefore unclear what SCA gains by restricting attention to fixed local neighborhoods between adjacent stages. Without a controlled comparison, it is difficult to separate the benefit of this choice from the benefits of gating and convolutional refinement.
The relationship to similar feature-pyramid methods needs more discussion. CFPT (IEEE TGRS 2025) also uses structured attention for efficient cross-layer interaction and incorporates positional relationships across layers. Its grouped channel-wise and spatial-wise attention differs from SCA’s sequential fusion between adjacent levels. Still, discussing these similarities and differences would help clarify what SCA adds beyond existing approaches.
Questions for authors
-
How do the diagnostics support the chosen window sizes, especially for Swin-T? It would help to show whether these windows cover the relevant cross-stage information and how performance changes with window size.
-
How does SCA compare with deformable attention, particularly Lite DETR’s key-aware variant, when the backbone, fusion topology, gating, and convolutional refinement are kept the same? Comparing accuracy and latency under these conditions would help clarify the benefit of fixed versus learned sampling locations.
2 comments
Response to Question 2: Controlled Comparison with Key-Aware Deformable Attention
Thank you for suggesting this controlled comparison. We compared SCA with a key-aware deformable attention variant based on Lite DETR (Li et al., 2023) while retaining the same backbone, bidirectional fusion topology, gating, and convolutional refinement. Appendix D.3 reports the comparison and its experimental settings.
SCA achieves 48.59 AP at 14.70 ms/image, compared with 46.60 AP at 16.46 ms/image for the key-aware deformable variant. This corresponds to 1.99 higher AP and 10.7% lower latency.
The comparison shows that fixed local retrieval can achieve both higher accuracy and lower inference cost than the evaluated learned-sampling alternative within this fusion architecture. Because both variants retain gating and convolutional refinement, the observed advantage supports the choice of the SCA attention mechanism within the shared design.
Response to Question 1: Diagnostic Support for Window Sizes and Cross-Stage Retrieval
Thank you for asking us to clarify how the diagnostics support the window design, particularly for Swin-T. We have expanded Appendix D with the direction-specific support rationale and window-size comparisons, and clarified the interpretation of the diagnostics in Section 4.3 and Appendix B.4.
Diagnostic motivation and window design. Swin-T’s impulse-response displacements motivate access to neighboring source positions around the geometric correspondence. For top-down exchange, the median displacements at S4 and S5 correspond to approximately 1.33 and 0.89 source tokens. A 3×3 window on these coarse grids spans 32 and 64 input pixels between its outer sample centers, respectively. Moreover, the updates are sequential: S4 incorporates S5 context before supplying S3. Thus, the immediate window is compact, while broader context accumulates across levels.
For bottom-up exchange, the coarse target maps halfway between fine-grid tokens, motivating symmetric sampling around that correspondence. Centered 4×4 support provides offsets of ±0.5 and ±1.5 source tokens along each axis. The measured displacement scale at the fine source levels, approximately 1.18–1.33 tokens, further motivates exploring surrounding positions.
These observations provide a spatial rationale for local exchange, but centroid displacement does not directly measure cross-stage correspondence or establish how much relevant information a window captures. Likewise, lower cross-stage similarity can reflect both displacement and stage-specific representations. We therefore use the diagnostics to motivate the design and assess the selected dimensions empirically.
Performance across window sizes. Appendix D.2 reports the following epoch-12 EMA results:
The selected configuration achieves nearly identical overall AP to 3×3/5×5, with higher AP-small and 36% fewer bottom-up sampling slots. Enlarging both windows to 5×5 gives higher overall AP, but lower AP-small and higher measured inference latency and memory use. These results support 3×3 top-down and centered 4×4 bottom-up support as a practical balance of overall accuracy, small-object performance, and resource requirements.