SCA-DETR: Structured Cross-Scale Attention for Detection Transformers with Hierarchical Backbones

Detection Transformers are a leading paradigm for end-to-end object detection, using learned object queries to attend to encoded image features and directly produce a set of detections. With hierarchical backbones, these detectors must integrate multi-scale features whose resolution, locality, and semantics differ across stages, making cross-scale interaction difficult to structure efficiently. Existing encoder designs face a trade-off between efficient local convolutional mixing, which restricts the scope of cross-scale exchange, and more expressive dense multi-scale attention, which incurs greater computational and memory cost. We introduce SCA-DETR, a Detection Transformer that uses Structured Cross-Scale Attention to organize information exchange across hierarchical backbone stages. To determine how this exchange should be structured, we use alignment diagnostics to distinguish geometric support from residual representational mismatch across adjacent backbone stages. Guided by this analysis, SCA uses the deterministic scale relationship between adjacent stages to center a local source neighborhood for each target location, constraining where cross-scale exchange occurs. Within this support, structured cross-scale attention determines what information to transfer, followed by residual refinement as the updates propagate in both directions through the hierarchy. On COCO val2017, SCA-DETR reaches 56.08 AP with TinyViT under the DEIM pipeline. With Swin-T, SCA improves the corresponding RT-DETR and DEIM-D-FINE baselines by 0.71 and 0.75 AP, respectively. Under compiled FP32 inference on an A100 GPU, the evaluated SCA configurations achieve 11–15% lower model-forward latency than their corresponding reference implementations. Additional results across hierarchical backbone families and on VisDrone2019-DET and PASCAL VOC support its applicability across architectures and detection settings.