acceptodds
Under review as a conference paper at ICLR 2027

SCA-DETR: Structured Cross-Scale Attention for Detection Transformers with Hierarchical Backbones

Abstract

Detection Transformers are a leading paradigm for end-to-end object detection, using learned object queries to attend to encoded image features and directly produce a set of detections. With hierarchical backbones, these detectors must integrate multi-scale features whose resolution, locality, and semantics differ across stages, making cross-scale interaction difficult to structure efficiently. Existing encoder designs face a trade-off between efficient local convolutional mixing, which restricts the scope of cross-scale exchange, and more expressive dense multi-scale attention, which incurs greater computational and memory cost. We introduce SCA-DETR, a Detection Transformer that uses Structured Cross-Scale Attention to organize information exchange across hierarchical backbone stages. To determine how this exchange should be structured, we use alignment diagnostics to distinguish geometric support from residual representational mismatch across adjacent backbone stages. Guided by this analysis, SCA uses the deterministic scale relationship between adjacent stages to center a local source neighborhood for each target location, constraining where cross-scale exchange occurs. Within this support, structured cross-scale attention determines what information to transfer, followed by residual refinement as the updates propagate in both directions through the hierarchy. On COCO val2017, SCA-DETR reaches 56.08 AP with TinyViT under the DEIM pipeline. With Swin-T, SCA improves the corresponding RT-DETR and DEIM-D-FINE baselines by 0.71 and 0.75 AP, respectively. Under compiled FP32 inference on an A100 GPU, the evaluated SCA configurations achieve 11–15% lower model-forward latency than their corresponding reference implementations. Additional results across hierarchical backbone families and on VisDrone2019-DET and PASCAL VOC support its applicability across architectures and detection settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Object detection, Detection transformers, Multi-scale fusion, Vision transformers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_N7cCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SCA-DETR, a method for fusing multi-scale features from hierarchical backbones in DETR-based detectors. SCA uses the scale relationship between adjacent stages to define local neighborhoods, then combines attention, gated fusion, and convolutional refinement along top-down and bottom-up paths. On COCO, the evaluated variants improve accuracy over matched RT-DETR and DEIM-D-FINE baselines while reducing latency under the reported inference settings.

Critical strengths

I think making more effective use of multi-scale features remains an important problem for DETR-based detectors. SCA offers a practical fusion module that works with several detector frameworks without changing their decoders or detection heads. The gains in accuracy and measured latency support its practical value.

Critical weaknesses

The diagnostics do not clearly explain why a small, fixed neighborhood is sufficient. Swin-T shows much larger impulse-response drift and lower cross-stage cosine similarity than ResNet-50. However, showing that the drift stays within an architecture-derived bound does not establish that SCA’s local windows cover the information needed for cross-stage fusion. Impulse-response drift does not directly measure cross-stage correspondence, and low cosine similarity may reflect differences in feature representations rather than spatial misalignment. The connection between these observations and the chosen window sizes needs more explanation.

The advantage over deformable attention is unclear. Deformable attention already gathers sparse cross-scale features around reference points, and Lite DETR’s key-aware variant also uses sampled keys to compute attention weights. It is therefore unclear what SCA gains by restricting attention to fixed local neighborhoods between adjacent stages. Without a controlled comparison, it is difficult to separate the benefit of this choice from the benefits of gating and convolutional refinement.

The relationship to similar feature-pyramid methods needs more discussion. CFPT (IEEE TGRS 2025) also uses structured attention for efficient cross-layer interaction and incorporates positional relationships across layers. Its grouped channel-wise and spatial-wise attention differs from SCA’s sequential fusion between adjacent levels. Still, discussing these similarities and differences would help clarify what SCA adds beyond existing approaches.

Questions for authors
  1. How do the diagnostics support the chosen window sizes, especially for Swin-T? It would help to show whether these windows cover the relevant cross-stage information and how performance changes with window size.

  2. How does SCA compare with deformable attention, particularly Lite DETR’s key-aware variant, when the backbone, fusion topology, gating, and convolutional refinement are kept the same? Comparing accuracy and latency under these conditions would help clarify the benefit of fixed versus learned sampling locations.

2 comments
AuthorsOpenReview ↗

Response to Question 1: Diagnostic Support for Window Sizes and Cross-Stage Retrieval

Thank you for asking us to clarify how the diagnostics support the window design, particularly for Swin-T. We have expanded Appendix D with the direction-specific support rationale and window-size comparisons, and clarified the interpretation of the diagnostics in Section 4.3 and Appendix B.4.

Diagnostic motivation and window design. Swin-T’s impulse-response displacements motivate access to neighboring source positions around the geometric correspondence. For top-down exchange, the median displacements at S4 and S5 correspond to approximately 1.33 and 0.89 source tokens. A 3×3 window on these coarse grids spans 32 and 64 input pixels between its outer sample centers, respectively. Moreover, the updates are sequential: S4 incorporates S5 context before supplying S3. Thus, the immediate window is compact, while broader context accumulates across levels.

For bottom-up exchange, the coarse target maps halfway between fine-grid tokens, motivating symmetric sampling around that correspondence. Centered 4×4 support provides offsets of ±0.5 and ±1.5 source tokens along each axis. The measured displacement scale at the fine source levels, approximately 1.18–1.33 tokens, further motivates exploring surrounding positions.

These observations provide a spatial rationale for local exchange, but centroid displacement does not directly measure cross-stage correspondence or establish how much relevant information a window captures. Likewise, lower cross-stage similarity can reflect both displacement and stage-specific representations. We therefore use the diagnostics to motivate the design and assess the selected dimensions empirically.

Performance across window sizes. Appendix D.2 reports the following epoch-12 EMA results:

Top-down supportBottom-up supportAPAP-small
3×33×3 floor-aligned48.1929.06
3×34×4 centered48.5929.94
3×35×5 floor-aligned48.6029.50
5×55×5 floor-aligned48.8329.28

The selected configuration achieves nearly identical overall AP to 3×3/5×5, with higher AP-small and 36% fewer bottom-up sampling slots. Enlarging both windows to 5×5 gives higher overall AP, but lower AP-small and higher measured inference latency and memory use. These results support 3×3 top-down and centered 4×4 bottom-up support as a practical balance of overall accuracy, small-object performance, and resource requirements.

AuthorsOpenReview ↗

Response to Question 2: Controlled Comparison with Key-Aware Deformable Attention

Thank you for suggesting this controlled comparison. We compared SCA with a key-aware deformable attention variant based on Lite DETR (Li et al., 2023) while retaining the same backbone, bidirectional fusion topology, gating, and convolutional refinement. Appendix D.3 reports the comparison and its experimental settings.

SCA achieves 48.59 AP at 14.70 ms/image, compared with 46.60 AP at 16.46 ms/image for the key-aware deformable variant. This corresponds to 1.99 higher AP and 10.7% lower latency.

The comparison shows that fixed local retrieval can achieve both higher accuracy and lower inference cost than the evaluated learned-sampling alternative within this fusion architecture. Because both variants retain gating and convolutional refinement, the observed advantage supports the choice of the SCA attention mechanism within the shared design.

Reviewer_7va6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SCA-DETR, an encoder-side multi-scale fusion method for Detection Transformers with hierarchical backbones. Motivated by cross-stage alignment diagnostics, the method exploits deterministic stride relationships to define local neighborhoods between adjacent feature levels. Within these neighborhoods, content-adaptive cross-scale attention enables information exchange, followed by gated fusion, residual refinement, and bidirectional hierarchy updates. The proposed module is integrated into RT-DETR, D-FINE, and DEIM-DFINE. Experiments on COCO val2017 report AP improvements of 0.51-0.96 points over corresponding baselines, alongside approximately 11-15% lower compiled FP32 modelforward latency in the evaluated configurations. Component ablations, cross-backbone evaluations, and additional experiments on VisDrone2019-DET and PASCAL VOC further assess the method's effectiveness and applicability.

Critical strengths
  1. Consistent Accuracy Efficiency Improvements. The paper evaluates SCA within matched RT-DETR, D-FINE, and DEIM-D-FINE pipelines, reporting consistent AP improvements of 0.51-0.96 points on COCO. The evaluated configurations also achieve approximately 11-15% lower compiled FP32 model-forward latency. These results demonstrate the practical value of the proposed fusion module.
  2. Comprehensive Experimental Validation. The paper provides extensive component ablations, alignment diagnostics, and local-support analyses to motivate its architectural choices. Evaluations across multiple hierarchical backbones and additional detection datasets further examine the method's applicability. These experiments provide meaningful evidence that the reported improvements are not limited to a single detector configuration.
Critical weaknesses
  1. Limited Coverage of Recent Related Work and Comparisons. Although the paper evaluates recent detector pipelines such as D-FINE and DEIM, the related work provides limited discussion of recent advances in efficient multi-scale fusion and cross-scale attention, particularly from 2025 to 2026. Moreover, the controlled comparisons primarily focus on established encoder baselines rather than alternative fusion methods. This makes it difficult to fully assess the proposed method's novelty and incremental contributions relative to recent developments.
  2. Insufficient Isolation of the Core Mechanism. While the alignment diagnostics and window-size ablations motivate geometry-guided local attention, they do not isolate the contribution of deterministic geometric anchoring. Moreover, the ablations do not independently evaluate spatial gating or compare cross-scale attention against simpler local fusion alternatives under comparable architectural and computational settings, leaving the benefits of the core design choices insufficiently established.
Questions for authors
  1. Relation to Recent Fusion Methods. Could the authors clarify the key differences between SCA and recent efficient multi-scale fusion or cross-scale attention methods, particularly from 2025 to 2026? A more focused discussion and comparison would help establish the proposed method's incremental contribution.
  2. Role of Core Components. Could the authors clarify the independent contributions of deterministic geometric anchoring and spatial gating? Would comparisons with simpler local fusion alternatives under similar architectural and computational settings help establish whether structured cross-scale attention is essential to the observed improvements?
2 comments
AuthorsOpenReview ↗

Response to Question 1: Relationship to Recent Multi-Scale Fusion Methods

Thank you for encouraging a broader discussion of recent multi-scale fusion methods. We have expanded the Multi-scale encoding and fusion discussion in Related Work to cover FPT (Zhang et al., 2020), CFPT, Speedy-DETR, and NeXt-DETR and to clarify how SCA’s fixed source neighborhoods differ from Lite DETR’s learned sampling locations (Li et al., 2023). Speedy-DETR attends to selected tokens from multiple pyramid levels and, in a parallel branch, to features within object-defined windows (Du et al., 2025a). NeXt-DETR shares the top-down and bottom-up fusion pattern of the RT-DETR HybridEncoder (Zhao et al., 2024) used in our comparisons. Both resize and concatenate feature maps across levels. NeXt-DETR’s FuNeXt module applies convolutional attention to the combined features (Choi & Lee, 2026). Neither Speedy-DETR nor NeXt-DETR explicitly maps each target location to a local neighborhood in one adjacent source level for query-dependent weighting. CFPT is a closer comparison because it explicitly models correspondence across pyramid levels (Du et al., 2025b).

CFPT first applies Cross-layer Channel-wise Attention (CCA) and then Cross-layer Spatial-wise Attention (CSA). In both stages, queries at each level attend to keys and values from corresponding token groups across all participating levels, with relative positional encoding accounting for their cross-level relationships. CFPT mixes channels across levels before its spatial attention stage can account for misalignment within a group. Once CSA reweights those positions, the initial cross-level channel update has already taken place. In our diagnostics, the hierarchical ViT showed greater spatial misalignment compared to the CNN backbone.

The spatial misalignment we observed in the hierarchical ViT motivated us to account for it during cross-level retrieval. SCA updates one target level at a time, attending to a local neighborhood in one adjacent source level. The target can therefore give greater weight to a nearby feature when it matches better than the feature at the nominal position. CFPT uses both channel-wise and spatial-wise attention for cross-level exchange. SCA makes its spatial selection during the initial lookup, uses a learned gate to control how much of the retrieved response enters the target, and then refines the target with cheaper depthwise and pointwise convolutions before the next cross-level update.

SCA builds on design principles reflected in recent fusion methods, including content-dependent attention in Speedy-DETR, bidirectional propagation in RT-DETR’s HybridEncoder and NeXt-DETR, and geometry-guided cross-level interaction in CFPT. SCA’s contribution is to organize fusion as a sequence of adjacent-level updates in which each target attends to a geometry-defined source neighborhood, gates the retrieved response, and becomes a refined source for the next exchange.

AuthorsOpenReview ↗

Response to Question 2: Contributions of Geometric Anchoring, Spatial Gating, and Content-Dependent Attention

Thank you for asking us to clarify SCA’s core components. Section 4.5 and Appendix E now discuss local aggregation and gated integration, with accuracy and parameter comparisons.

Deterministic geometric anchoring. Adjacent pyramid levels represent the image on grids with different spatial resolutions. To gather a local source neighborhood for a target location, SCA first needs to determine where that neighborhood lies on the source grid. The known stride relationship between the levels provides this geometric anchor. The local support specifies which neighboring features participate, and attention allows the target to assign greater weight to features that provide a better match.

This mapping is integral to defining the local retrieval operation. Uniform aggregation and learned positional weighting also need a source neighborhood over which to operate. Both therefore retain the same anchor, allowing the aggregation comparison to examine different weighting rules within a common neighborhood.

RT-DETR’s HybridEncoder establishes a common grid through upsampling or strided convolution before concatenation and convolutional fusion. If corresponding feature responses remain spatially offset after resizing, features at the same grid position may represent different spatial content. SCA uses its geometric anchor to identify a local source neighborhood and computes target-dependent weights over the positions within it during retrieval. The following comparison evaluates the complete fusion designs under the 12-epoch Swin-T setting.

Fusion approachCross-level interactionAPDetector parameters
RT-DETR HybridEncoderResize, concatenate, and convolutionally fuse features47.4946.36M
SCARetrieve from mapped source neighborhoods, then gate and refine48.6644.61M

In the 12-epoch Swin-T comparison, the detector with SCA achieved 1.17 AP more than the detector with RT-DETR’s HybridEncoder while using 1.75M fewer parameters.

Spatial gating. Attention selects information from the source neighborhood, while gating regulates its integration with the target. The gate is predicted jointly from the target feature and the retrieved response, allowing their relative contributions to vary with local content. Both variants use content-dependent cross-scale attention, allowing us to compare additive and spatially gated integration of the retrieved response. Appendix E.2, Table S13, reports this comparison, summarized below.

IntegrationAPDetector parameters
Additive fusion48.3943.56M
Spatially gated fusion48.6644.61M

The observed 0.27 AP improvement supports adaptive integration of cross-scale information. This comparison uses a separate support configuration from the attention comparison below, and the two gains describe their respective experimental settings.

Content-dependent attention. Geometric anchoring identifies a local source neighborhood, but the features within it need not be equally relevant to every target. Corresponding responses may occur at neighboring positions, and the most useful source feature can vary with the image content. Content-dependent attention allows each target to weight those features according to their compatibility with its query. We evaluate this choice against uniform aggregation and learned positional weighting while retaining the same neighborhoods, bidirectional update paths, gating, refinement, and value/output projections. Uniform aggregation assigns equal weights to sampled positions, while learned positional weighting learns weights shared across images. Under the 12-epoch, seed-0 Swin-T protocol on COCO, Appendix E.1, Table S12, reports the aggregation results, summarized below.

Local aggregationAPDetector parameters
Uniform aggregation48.2944.08M
Learned positional weighting48.4244.08M
Content-dependent attention48.6644.61M

Both uniform aggregation and learned positional weighting outperform HybridEncoder while using fewer parameters. These variants retain geometrically defined neighborhoods, gated integration, and sequential refinement, supporting this shared organization of cross-level exchange. Spatial gating adds 0.27 AP in its comparison, while content-dependent attention achieves the highest AP among the tested aggregation rules. These results support structured local fusion, with adaptive weighting and integration further improving how information is selected and incorporated.

Reviewer_LwKZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses cross-scale feature fusion in hierarchical backbones for object detection and proposes Structured Cross-Scale Attention (SCA), a module designed to replace the encoder-based fusion component in DETR-style detectors. The key idea is to exploit the spatial relationships between adjacent feature levels to establish structured cross-scale interactions, enabling content-dependent information exchange within local windows while reducing unnecessary computation.  The main contributions of the paper include: 

  1. Structured Cross-Scale Attention. The proposed SCA module uses the stride relationships between adjacent feature levels to determine source feature locations and performs content-dependent cross-attention within local windows. It further incorporates gated fusion and depthwise-separable convolutional residual refinement to improve cross-scale feature integration. 
  2. Bidirectional Hierarchical Feature Propagation. The method adopts a top-down and bottom-up propagation scheme, using 3×3 and centered 4×4 windows, respectively. It retains the original detector decoder and prediction heads, facilitating integration into existing detection architectures. 
  3. Diagnostic Analysis for Design Motivation. The paper investigates impulse-response displacement and cross-level feature consistency to motivate its geometric alignment and learned information exchange designs. 
  4. Evaluation Across Detection Frameworks and Backbones. The method is evaluated on RT-DETR, D-FINE, and DEIM-D-FINE with different hierarchical backbones. On COCO val2017, the four reported paired comparisons show AP improvements ranging from 0.51 to 0.96 points. The paper also provides component ablations, experiments involving five hierarchical backbones, additional evaluations on VisDrone and VOC, and inference-cost measurements on an A100 GPU.  Overall, the reported results suggest that SCA can improve detection accuracy while reducing inference latency and memory consumption under the evaluated settings. However, the extent to which these improvements can be attributed specifically to the proposed attention mechanism, rather than to the broader fusion architecture and refinement components, remains insufficiently established.
Critical strengths
  1. The proposed method addresses a practically relevant problem in object detection. Cross-scale feature fusion is an important component of detection transformers with hierarchical backbones, where feature representations at different resolutions must be effectively integrated. The proposed SCA module provides a structured approach to cross-level information exchange while retaining the original decoder and prediction heads. This design makes the method potentially useful for improving existing detection systems without requiring substantial changes to their downstream components. 
  2. Paired experiments demonstrate consistent improvements across multiple detector configurations. The four main experiments report positive AP improvements when SCA is incorporated into the corresponding baseline configurations, covering RT-DETR, D-FINE, and DEIM-D-FINE. The reported gains range from 0.51 to 0.96 AP points. The additional comparisons across five hierarchical backbones also reportedly show positive improvements. These paired results provide more meaningful evidence of the module's effectiveness than comparisons against unrelated methods with different backbones or training configurations. 
  3. The method demonstrates promising inference-efficiency improvements alongside accuracy gains. Under the reported A100, batch-size-1, compiled FP32 inference setting, RT-DETR with Swin-T reduces latency from 17.383 ms to 14.697 ms and peak allocated memory from 342.23 MiB to 185.69 MiB. The paper also reports latency reductions of approximately 11% for two DEIM configurations. These measurements suggest that SCA may provide practical efficiency benefits in addition to improved detection accuracy, although the conclusions should remain limited to the evaluated hardware and inference conditions.
Critical weaknesses
  1. The novelty and independent contribution of the proposed attention mechanism remain insufficiently established. The paper introduces structured local cross-scale attention, bidirectional feature propagation, gated fusion, and convolutional refinement. However, the distinction between the proposed design and existing cross-scale attention methods remains insufficiently clarified. In particular, prior approaches such as FPT and CFPT already investigate cross-level transformer interactions, local spatial attention, and scale-aware positional modeling. The paper needs to more precisely identify which aspects of SCA constitute genuinely new technical contributions and how they differ from these closely related approaches. More importantly, the current ablations do not fully isolate the contribution of content-dependent attention. The reported improvements arise from a combination of cross-level information exchange, bidirectional propagation, and feature refinement. Without a controlled comparison that preserves the same bidirectional paths, gating mechanisms, and refinement capacity while replacing content-dependent attention with fixed local aggregation or conventional feature fusion, it remains unclear whether the proposed attention mechanism itself provides a substantial advantage. This limits the strength of the paper's central methodological claim.

 2. The diagnostic analyses provide insufficient evidence for the proposed design rationale. The paper uses impulse-response displacement and cross-level feature similarity to motivate its geometric alignment and local-window design. However, the interpretation of these diagnostics requires further clarification. In particular, the definitions of global and local cosine similarity and normalized cross-correlation (NCC), as well as the handling of channel dimensions, feature projections, normalization, and local search ranges, need to be specified more precisely. Without ensuring that the compared representations are meaningfully aligned, low cross-level cosine similarity cannot necessarily be attributed to semantic misalignment. Furthermore, the spatial displacement of impulse-response centroids does not directly establish the correspondence between adjacent-level tokens or guarantee that the selected attention windows capture the relevant cross-scale responses. The diagnostic results therefore provide suggestive motivation but do not yet sufficiently demonstrate that the proposed window sizes and alignment strategy are uniquely justified by the observed feature geometry. More direct evidence connecting the diagnostics to actual cross-level correspondence and window coverage would strengthen the methodological rationale.

 3. The reliability of the reported improvements requires stronger experimental controls. Although all four main paired comparisons report positive AP gains, the improvements are relatively modest, ranging from 0.51 to 0.96 points. The reported ablations primarily use a 12-epoch setting with a single random seed, and the main experiments do not appear to provide systematic multi-seed statistics. Consequently, the variability of the reported improvements remains unclear. In addition, the fairness of the paired comparisons depends on whether the baseline and SCA-equipped models use identical pretrained weights, training schedules, optimization settings, and checkpoint-selection procedures. These details are important for determining whether the observed gains can be attributed to SCA rather than differences in training or model selection. More explicit experimental controls and representative multi-seed results would make the conclusions more convincing.

Questions for authors
  1. What is the independent contribution of SCA relative to existing cross-scale attention methods? How does SCA differ technically from closely related approaches such as CFPT, particularly in its cross-scale interaction patterns, local attention formulation, and positional modeling? More importantly, when the backbone, bidirectional propagation paths, gating mechanisms, and refinement modules are kept unchanged, does content-dependent SCA provide a measurable advantage over fixed local aggregation or conventional feature fusion? A direct controlled comparison would help establish whether the proposed attention mechanism itself is responsible for the reported improvements. 
  2. How are the diagnostic metrics defined, and how directly do they justify the proposed window design? Could the authors provide the complete computation protocols for global/local cosine similarity and NCC, including channel matching, feature projection, normalization, and local search ranges? Furthermore, is there direct evidence that the selected windows cover meaningful cross-level responses and that the diagnostic findings remain valid after controlling for differences in feature statistics? These clarifications would materially affect the assessment of the paper's diagnostic analysis and its role in motivating the SCA design. 
  3. Are the reported AP improvements robust under strictly matched experimental settings? Do the baseline and SCA-equipped models use identical pretrained weights, training schedules, optimization settings, and EMA checkpoint-selection rules? Could the authors provide a more explicit account of these paired experimental settings and report mean AP and variability across multiple random seeds for at least one representative main experiment? This would help determine whether the reported improvements of 0.51–0.96 AP points are robust and attributable to the proposed method rather than experimental variation.
3 comments
AuthorsOpenReview ↗

Response to Question 1: Technical Novelty and Contribution of Content-Dependent Attention

Thank you for asking us to clarify SCA's technical contribution and the independent benefit of content-dependent attention. We have expanded Related Work and updated Section 4.5 and Appendix E with the local aggregation results. SCA organizes cross-scale exchange as sequential updates between adjacent levels. Each target retrieves information from a geometry-defined local source neighborhood, gates the response, and refines the updated feature before subsequent exchanges.

Relationship to FPT. In FPT’s detection configuration, top-down grounding uses global cross-level attention: each fine-level query is compared with all coarse-level source positions. Its attention cost grows with both feature-map resolutions, and distant features compete with nearby candidates. SCA restricts retrieval to a geometry-defined neighborhood and uses content-dependent attention to select among nearby positions, accommodating local displacement within a bounded candidate set. FPT’s bottom-up rendering combines globally pooled channel weighting with strided convolution, without target-specific attention over source positions. SCA applies local content-dependent retrieval in both directions.

Relationship to CFPT. CFPT first applies Cross-layer Channel-wise Attention (CCA), followed by Cross-layer Spatial-wise Attention (CSA). Both exchange information across corresponding token groups from the participating levels, with relative positional encoding representing cross-level relationships. CFPT therefore performs its initial cross-level channel update before CSA explicitly reweights source positions.

SCA makes spatial selection part of the initial retrieval. Each target attends to a geometry-defined neighborhood in one adjacent source level, allowing a nearby feature to receive greater weight when it matches better than the nominally corresponding feature. This selects information before integration with the target. A learned spatial gate controls the retrieved contribution, followed by depthwise and pointwise convolutional refinement. Thus, CFPT applies spatial selection after cross-layer channel attention, while SCA applies it during source retrieval, before gated integration and refinement.

Sequential propagation. FPT combines separately transformed features, while CFPT performs grouped cross-layer interactions through channel and spatial attention. SCA updates adjacent levels sequentially, using features enriched by preceding exchanges. Its bottom-up pass therefore builds on context accumulated during top-down propagation.

Positional modeling. CFPT uses CCPE to encode relative positions within cross-layer token groups through a shared coordinate system and interpolated position biases. SCA’s adjacent-grid mapping directly defines the local source neighborhood. Learned per-offset biases encode spatial preferences within it, while query–key attention adapts the weights to image content.

Independent contribution of content-dependent attention. We compare three aggregation rules while retaining the Swin-T backbone, source neighborhoods, bidirectional update paths, gating, refinement, and value/output projections. Uniform aggregation assigns equal source weights. Learned positional weighting learns spatial preferences shared across images. Content-dependent attention adds query–key compatibility, making the weights depend on target and source features.

The 12-epoch, seed-0 RT-DETR/Swin-T results on COCO val2017 are reported in Appendix E. We include HybridEncoder as a complete-fusion reference; the three SCA rows retain the shared structure described above.

Fusion / aggregation variantAPDetector parameters
RT-DETR HybridEncoder47.4946.36M
SCA — uniform aggregation48.2944.08M
SCA — learned positional weighting48.4244.08M
SCA — content-dependent attention48.6644.61M

Content-dependent attention improves AP by 0.37 over uniform aggregation and 0.24 over learned positional weighting. Both positional weighting and attention learn spatial preferences within the same support, while attention additionally adapts source weights to image content.

Both simpler variants outperform HybridEncoder, supporting the shared design of geometrically structured local retrieval, gated integration, and sequential refinement. Content-dependent attention further improves AP within this structure. The observed gains therefore reflect the combination of structured cross-scale exchange and adaptive weighting.

AuthorsOpenReview ↗

Response to Question 2: Diagnostic Definitions and Window-Size Rationale

Thank you for asking us to clarify the diagnostic calculations and how they inform the window design. We have expanded Appendix B.4 with the complete computation protocol and Appendix D with the direction-specific support rationale and window-size comparisons. We have also clarified the interpretation of the diagnostics in Section 4.3.

Diagnostic definitions and feature statistics. Cosine similarity and NCC compare spatial patterns in channel-reduced activation maps. At each spatial location, we subtract the mean across channels and take the L2 magnitude of the remaining responses. This produces one scalar map per stage, without learned projection or channel-by-channel matching. We then resample the maps to a common grid and standardize them independently. Global cosine compares the complete maps, while local cosine compares spatial patches. NCC additionally centers each patch and searches nearby positions for matching patterns. Appendix B.4 specifies the patch sizes, search ranges, and aggregation rules.

Under this normalized comparison, Swin-T’s local NCC is 0.2724 and 0.3369 for S4 → S3 and S5 → S4, compared with 0.6545 and 0.4667 for ResNet-50. These differences remain after channel centering, spatial standardization, and local mean-centered matching. They describe differences in spatial activation patterns, which can reflect both displacement and stage-specific representations.

How the diagnostics inform the windows. Swin-T’s impulse-response displacements motivate access to neighboring source positions around the geometric correspondence. For top-down exchange, the median displacements at S4 and S5 correspond to approximately 1.33 and 0.89 source tokens. A small window on these coarse grids spans a substantial input-image region, and the sequential updates allow S3 to receive S5 context through the updated S4 features. These considerations motivate compact 3×3 top-down support.

For bottom-up exchange, the coarse target maps halfway between fine-grid tokens, motivating symmetric sampling around that correspondence. Centered 4×4 support provides offsets of ±0.5 and ±1.5 source tokens along each axis. The measured displacement scale at the fine source levels, approximately 1.18–1.33 tokens, further motivates exploring surrounding positions.

Centroid displacement does not directly measure cross-stage correspondence or establish how much relevant information a window captures. We therefore use the diagnostics to motivate the design and assess the selected dimensions empirically.

Performance across window sizes. Appendix D.2 reports the following epoch-12 EMA results:

Top-down supportBottom-up supportAPAP-small
3×33×3 floor-aligned48.1929.06
3×34×4 centered48.5929.94
3×35×5 floor-aligned48.6029.50
5×55×5 floor-aligned48.8329.28

The selected configuration achieves nearly identical overall AP to 3×3/5×5, with higher AP-small and 36% fewer bottom-up sampling slots. The larger 5×5/5×5 configuration gives higher overall AP, but lower AP-small and higher measured inference latency and memory use. These results support the selected windows as a practical balance of overall accuracy, small-object performance, and resource requirements.

AuthorsOpenReview ↗

Response to Question 3: Matched Experimental Settings and Seed Variability

We agree that multi-seed replication is needed to quantify training variability. The reported comparisons use a single seed, and we could not complete a matched, full-schedule multi-seed study within the rebuttal period.

Within each detector–backbone pair, the HybridEncoder and SCA configurations use the same pretrained backbone weights, training schedule, optimizer settings, data augmentation, and EMA checkpoint-selection rule. These shared settings, detailed in Appendix A.1, control the training and model-selection factors identified by the reviewer.

Under these paired settings, SCA improves AP in all four evaluated configurations: +0.71 for RT-DETR/Swin-T, +0.51 for D-FINE/Swin-T, +0.75 for DEIM-D-FINE/Swin-T, and +0.96 for DEIM-D-FINE/TinyViT. The improvement is consistently observed across three detector pipelines and two backbone families, despite differences in their architectures and training recipes. These results demonstrate a consistent empirical benefit across the evaluated configurations and support SCA’s applicability as a fusion replacement across multiple detection systems.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.