Manifold-DETR: Local Manifold-Aligned Attention for Object Detection

Sparse detection transformers reduce encoder computation by updating selected image tokens, but allocating updates according to aggregate attention demand can favor already well-represented regions. Effective sparse detection also depends on how queries read local features and determine object boundaries. We present Manifold-DETR, a detection framework that aligns local feature gathering and sparse update allocation with the detection task. Local Manifold-Aligned Attention (LMAA) uses a direction–magnitude descriptor and its induced local geometry to correct query-predicted sampling locations, then evaluates content compatibility to aggregate the sampled features. Joint Boundary Refinement combines localization-specific boundary features with the query state to predict four coordinated box-side residuals. A task-calibrated decoder attention map (TDAM) uses matched-instance demand to supervise encoder token selection, with restricted reranking that preserves a high-demand core. On COCO, Manifold-DETR improves over Sparse-DETR by 2.1–2.8 AP with ResNet-50 across encoder keeping ratios of 10%–50% after 50 training epochs. At a 10% keeping ratio, it achieves 48.9 AP with ResNet-50 and 50.8 AP with Swin-T, surpassing the corresponding Sparse-DETR models at 50% while updating one fifth as many encoder tokens. These results demonstrate the value of task-aligned local feature reading and update allocation under sparse encoder budgets.