acceptodds
Under review as a conference paper at ICLR 2027

Manifold-DETR: Local Manifold-Aligned Attention for Object Detection

Abstract

Sparse detection transformers reduce encoder computation by updating selected image tokens, but allocating updates according to aggregate attention demand can favor already well-represented regions. Effective sparse detection also depends on how queries read local features and determine object boundaries. We present Manifold-DETR, a detection framework that aligns local feature gathering and sparse update allocation with the detection task. Local Manifold-Aligned Attention (LMAA) uses a direction–magnitude descriptor and its induced local geometry to correct query-predicted sampling locations, then evaluates content compatibility to aggregate the sampled features. Joint Boundary Refinement combines localization-specific boundary features with the query state to predict four coordinated box-side residuals. A task-calibrated decoder attention map (TDAM) uses matched-instance demand to supervise encoder token selection, with restricted reranking that preserves a high-demand core. On COCO, Manifold-DETR improves over Sparse-DETR by 2.1–2.8 AP with ResNet-50 across encoder keeping ratios of 10%–50% after 50 training epochs. At a 10% keeping ratio, it achieves 48.9 AP with ResNet-50 and 50.8 AP with Swin-T, surpassing the corresponding Sparse-DETR models at 50% while updating one fifth as many encoder tokens. These results demonstrate the value of task-aligned local feature reading and update allocation under sparse encoder budgets.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Object Detection, Deformable Attention, Local Manifold Geometry, Adaptive Sampling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_L4izOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Manifold-DETR, an object detection framework built on Sparse-DETR to improve detection accuracy under limited encoder token updates. It introduces three components, first Local Manifold-Aligned Attention (LMAA): Uses local feature geometry to correct deformable attention sampling locations and improve feature aggregation, second Joint Boundary Refinement (JBR): Samples features along predicted bounding box boundaries to jointly refine all four sides and last Task-Calibrated Decoder Attention Map (TDAM): Uses instance-level attention from matched object queries to improve encoder token selection during training. Experiments on COCO with ResNet-50 and Swin-T show consistent improvements over Sparse-DETR across different encoder keeping ratios. Ablation studies evaluate the contributions of each component.

Critical strengths
  • Clear motivation: The paper identifies an important limitation of Sparse-DETR: attention demand does not necessarily reflect where additional encoder computation is most useful.
  • Interesting attention improvement: LMAA addresses a limitation of deformable attention by using local feature geometry to refine query-predicted sampling locations rather than relying entirely on predicted offsets.
  • No inference overhead from TDAM: The proposed task-calibrated supervision improves token selection without adding an additional inference-time branch.
  • Good ablation studies: The experiments reasonably isolate the contributions of the proposed components and demonstrate consistent improvements over Sparse-DETR across different token keeping ratios.
Critical weaknesses
  • Outdated baseline and limited comparisons: The framework is built on Sparse-DETR, an older detector. Although Table 1 is presented as a comparison with state-of-the-art detectors, it lacks stronger recent architectures such as DINO, RT-DETR, and D-FINE. This limits the evaluation of the method's competitiveness and generalizability.
  • Limited dataset evaluation: Experiments are conducted only on COCO. Evaluation on additional datasets, particularly those with small or difficult objects, would better support the proposed motivation.
  • Significant inference overhead: Although Manifold-DETR improves accuracy with fewer updated encoder tokens, its inference latency increases substantially. At a 10% keeping ratio, latency increases from 27.18 ms to 62.97 ms compared with Sparse-DETR, despite a negligible increase in FLOPs. This weakens the efficiency claims.
  • Unequal contribution of components: LMAA provides most of the improvement, while JBR and TDAM offer relatively small additional gains. Removing JBR reduces AP by only 0.2, while removing TDAM reduces it by 0.4.
  • Limited effectiveness across decoder layers: LMAA performs best when applied only to the final decoder layer. Applying it to more layers reduces accuracy, raising questions about its limited effectiveness when extended to earlier decoder layers.
Questions for authors
  • Q1 Can LMAA be integrated into newer architectures such as DINO, RT-DETR, and D-FINE ?
  • Q2 Can the authors evaluate the method on additional datasets, particularly those containing small, difficult, or weakly attended objects?
  • Q3 How do the proposed components behave with different numbers of object queries, such as 100, 300, and 900?
  • Q4 Why does inference latency increase substantially despite the negligible increase in FLOPs? Which operations introduce the largest overhead?
  • Q5 Why does applying LMAA to additional decoder layers reduce performance?
2 comments
AuthorsOpenReview ↗

LMAA improves four modern detectors and two aerial datasets, with measured inference overhead.

Thank you for the careful review. We have added experiments on four detection frameworks and two aerial datasets, repeated the key component comparisons, and expanded the query-count, timing, and feature-reading analyses.

Response to W1 and Q1: LMAA in modern detection frameworks

LMAA improves DINO, D-FINE, Lite DETR, and RT-DETR. Appendix E.5, Table 19, compares each framework's original decoder attention with its LMAA adaptation. Each pair uses ResNet-50 and the same 12-epoch COCO schedule under the corresponding framework configuration.

FrameworkAP: original → LMAAAP_75AP_s
DINO49.3 → 50.653.8 → 55.832.5 → 33.8
D-FINE50.2 → 51.554.5 → 56.432.2 → 34.0
Lite DETR48.2 → 49.652.4 → 53.530.3 → 31.3
RT-DETR47.3 → 48.251.3 → 52.328.1 → 29.0

The AP improvements are 1.3, 1.3, 1.4, and 0.9 points, with gains in AP_75 and AP_s in all four pairs. D-FINE is particularly informative: its stronger localization design still benefits from improved feature reading, reaching 51.5 AP and 56.4 AP_75. Lite DETR tests another attention architecture, while DINO and RT-DETR extend the comparison to different detector configurations.

LMAA operates on a query, a spatial reference, and multiscale memory. This interface allows its geometric correction and content aggregation to be integrated independently of the Sparse-DETR selector and TDAM supervision.

Response to W2 and Q2: Additional datasets with small and difficult objects

Appendix E.6 adds AI-TOD and DOTA, comparing Sparse-DETR with the complete Manifold-DETR. Both use ResNet-50 and ρ = 0.1, with the same encoder update budget.

For AI-TOD, the models use 72,000 training updates, 11,214 training images, and 2,804 validation images. Evaluation follows the dataset's four size categories and allows at most 1,500 detections per image. Table 20 reports AP over IoU thresholds 0.50:0.05:0.95:

ModelAPAP_50AP_75AP_vtAP_tAP_sAP_m
Sparse-DETR19.245.811.48.819.122.327.8
Manifold-DETR21.648.814.711.220.325.229.9

Overall AP improves by 2.4 points, including a 2.4-point gain for very tiny objects. Table 21 reports AR for the very tiny, tiny, small, and medium groups: 14.6/33.9/36.5/38.6 for Sparse-DETR and 17.2/35.2/38.9/40.1 for Manifold-DETR.

For DOTA, Table 22 reports horizontal-box AP_50 on image tiles after 24 epochs, averaged over all 16 categories. Mean AP_50 increases from 58.5 to 60.4, and small-vehicle AP_50 improves from 34.6 to 37.7. The complete category results are provided in the table.

Response to W3 and Q4: Inference overhead despite a small FLOP increase

We repeated timing with the benchmarking GPU reserved to avoid interference from concurrent jobs on the shared server. Appendix E.3 specifies one RTX 4090, FP32, batch size one, and 800 × 1333 inputs. Timing covers inference and GPU box postprocessing, excluding I/O and data transfer. Each configuration receives 50 warmup passes and six groups of 50 timed passes with GPU synchronization. Table 14 reports the following mean latencies:

ImplementationLatency (ms)
MSDA baseline26.9
LMAA53.8
LMAA with validity checking from the mask47.3

Computing candidate validity from the mask reduces LMAA latency by approximately 6.5 ms, or 12%, while retaining the local geometric reading procedure.

Table 17 identifies the main bottlenecks of the original LMAA implementation. Initial sampling and spatial derivatives account for 8.3 ms of host interval and 1152 kernel launches per image; candidate validity accounts for 8.4 ms and 1164 launches. Together, these account for approximately 63% of the total host time and 66% of kernel launches reported in Table 17. Descriptor/Jacobian computation takes 2.8 ms of host interval, metric and linear-system construction 1.4 ms, and the bounded position solve 3.3 ms.

Local sampling, indexing, validity processing, and many small kernel launches explain the runtime overhead beyond the arithmetic counted by FLOPs. The mask implementation directly reduces one of the largest stages.

AuthorsOpenReview ↗

LMAA is the main contributor, the final decoder layer is the best placement, and gains hold at 100–900 queries.

Response to W4: The respective contributions of LMAA, JBR, and TDAM

LMAA is the principal contribution. JBR improves boundary localization, and TDAM calibrates encoder update supervision. The revision adds repeated comparisons and isolated measurements of these roles.

Table 11 repeats the comparison with JBR and TDAM fixed. Across three training runs, JBR+TDAM reaches 47.3 ± 0.2 AP, and the complete model reaches 49.0 ± 0.1 AP, a 1.7-point mean improvement from LMAA. AP_75 changes from 51.5 ± 0.3 to 52.7 ± 0.0. Values are means and sample standard deviations.

The design comparisons have also been repeated. Table 4 fixes LMAA and original DAM: independent side prediction gives 48.4 ± 0.2 AP and joint prediction gives 48.8 ± 0.2. Table 7 fixes LMAA and disables JBR: restricted random refill gives 47.9 ± 0.5 AP and TDAM gives 48.9 ± 0.4.

We also evaluate each auxiliary component without LMAA. For JBR, Tables 5–6 compare Sparse-DETR with and without boundary refinement. At ρ = 0.1, ResNet-50's mean matched-box IoU increases from 76.1 to 78.6 and Mean Normalized Absolute Edge Error decreases from 21.8 to 20.1. Swin-T gives corresponding changes of 82.3 → 86.7 and 19.1 → 15.4. Both measures are multiplied by 100; the edge error normalizes each side by GT width or height.

For TDAM, Tables 8–10 change only DAM supervision. At the same update ratio, small-object coverage by actual selected centers increases from 98.1% to 99.9% with ResNet-50 and from 98.8% to 100.0% with Swin-T. Small-object AR improves from 47.7 to 48.6 and from 48.4 to 50.4; isolated AP also improves from 46.8 to 47.3 and from 49.6 to 50.3. The paired visualizations in Appendix E.1.7 show the corresponding changes in the selected subsets.

Response to W5 and Q5: Why the final decoder layer is the best placement

Table 15 compares placements using ResNet-50, full COCO, 50 epochs, and ρ = 0.1. AP is 46.8 without LMAA, 47.9 with LMAA in the final layer, 47.3 in the final two layers, and 46.9 in all six layers. Our most plausible explanation is the interaction between early-query reliability and the local range of geometric correction.

LMAA aligns local features with the current query descriptor. For the i-th sampling point, Jᵢ is the descriptor Jacobian at the current iterate (initialized at xᵢ⁰), rᵢ is the residual between the current query descriptor and the target descriptor t, and λ is the damping weight:

vᵢ = −(JᵢᵀJᵢ + λI)⁻¹ Jᵢᵀrᵢ, rᵢ = zᵧ(k(xᵢ⁰)) − zᵧ(t)

The damped update is capped at 0.5 feature cell. Early queries and reference boxes may still be ambiguous, so improving a local match can reinforce an intermediate interpretation and affect subsequent query and box refinement. Repeating this correction across layers may compound that bias, since the target descriptor and sampling state change at each layer. Applying LMAA at the final layer lets local alignment operate after the preceding stages have refined the object representation and spatial reference.

Figure 16 illustrates this local behavior using 2741 sampling slots initially outside their matched boxes in four final-layer examples. For initial distances of 0–0.1 GT-box diagonals, 39.4–51.7% of slots enter the box; for distances of 0.1–0.5, the fraction is 17.1–35.0%. This pattern is consistent with our interpretation that bounded geometric correction is most useful once the sampling neighborhood is already close to the object.

Response to Q3: Different numbers of object queries

Appendix E.1.5, Table 12, evaluates 100, 300, and 900 queries. Each model is trained for 50 epochs on full COCO with ResNet-50 and ρ = 0.1, using the corresponding query count.

Number of queries100300900
Sparse-DETR AP45.946.848.2
Manifold-DETR AP48.448.949.8
AP improvement+2.5+2.1+1.6
Sparse-DETR AP_s26.528.830.4
Manifold-DETR AP_s28.229.931.5

Manifold-DETR improves AP and AP_s at all three query budgets. Absolute AP increases with more queries, while the AP gain over the matched baseline decreases from 2.5 to 1.6 points.

Reviewer_JkPrOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Manifold-DETR, which combines three components (LMAA, JBR and TDAM) within Sparse-DETR. Local Manifold-Aligned Attention (LMAA) uses a direction–magnitude descriptor and its spatial Jacobian to correct attention sampling locations. Joint Boundary Refinement (JBR) predicts four box-side residuals from boundary features. A task-calibrated decoder attention map (TDAM) modifies token-selection supervision while preserving the original high-demand core. Experiments on COCO cover two backbones and multiple encoder update ratios (10-50%). With ResNet-50 and Swin-T, 50 training epochs, and a 10% update ratio, the complete model achieves 48.9 and 50.8 AP, over Sparse-DETR with a 50% update ratio.

Critical strengths
  1. LMAA provides a valuable approach to local sampling refinement. It uses the sampled features and their spatial variation to correct query-predicted locations, coupling coordinate refinement with content-dependent aggregation.
  2. The overall accuracy gains over Sparse-DETR are reasonably consistent. In the 50-epoch experiments, the proposed model improves overall AP at every update ratio for both backbones.
  3. The ablation coverage and implementation details are substantial. The paper evaluates all seven component combinations and examines the position and content branches, LMAA placement, etc.
Critical weaknesses
  1. The overall AP improvement alone cannot sufficiently validate the core motivation concerning weakly represented targets. The introduction motivates the method through a mismatch between attention demand and update utility, particularly for small or weakly represented objects. In the Swin-T, 50-epoch experiments, AP_S at 20%–50% update ratios is consistently lower than that of Sparse-DETR. The small-object improvements with ResNet-50 and the shorter schedule are encouraging, but the results do not establish a general benefit for weakly represented small objects.
  2. The component contributions interact, and the small gains lack evidence of statistical stability. For example, in table 2, adding JBR or TDAM individually yields 47.5 and 47.3 AP, respectively, whereas their combination yields 47.1 AP, indicating that the reported contributions are not additive. In table 5, TDAM improves AP by only 0.3 compared with constrained random replacement. In table 4, joint boundary prediction improves AP by only 0.2 compared with independent prediction. The absence of means and standard deviations over multiple random seeds makes it difficult to judge the stability of these differences. The existing ablations support effectiveness under some configurations but are insufficient to confirm that every detail yields stable and reproducible gains.
  3. Reducing encoder update tokens does not produce a speedup in the current implementation. In table 7, adding LMAA only to the last decoder layer increases latency from 27.18 ms to 58.67 ms. Updating one fifth as many encoder tokens accurately describes the update budget but does not establish an end-to-end speed advantage. The authors have disclosed this limitation in Appendix E.5. The accuracy–latency trade-off needs to be discussed more fully in the main conclusions.
Questions for authors
  1. What explains the reductions in (AP_S) across several update ratios for Swin-T after 50 epochs? Can you isolate TDAM and report coverage by the actual predicted token subset and final detection recall, stratified by object size?
  2. Can you repeat the key comparisons across multiple training seeds and report the number of runs, means, and standard deviations, especially for TDAM versus restricted random refill, joint versus independent boundary prediction, and the JBR–TDAM combination?
  3. Which operations dominate LMAA’s latency? Results in tabel 7 show a great delay when applying 6 LMAA layers.
  4. Under matched backbone, training, and sampling-budget settings, how much does LMAA improve over Lite DETR-style key-aware attention or a simpler learned local-offset correction? Can this benefit be demonstrated within a stronger detector, such as a framework (like D-FINE) with recent localization refinements?
2 comments
AuthorsOpenReview ↗

TDAM alone improves coverage, recall and AP, including on Swin-T, confirmed by repeated runs.

Thank you for your comments. We have added isolated TDAM experiments measuring actual selections and recall, repeated training for the key design comparisons, both suggested alternative attention readers, and LMAA results in four additional detection frameworks.

Response to W1 and Q1: Swin-T small objects and the effect of TDAM

With Swin-T and 50 training epochs, the complete model has lower AP_s than Sparse-DETR at keeping ratios of 20%, 30%, 40%, and 50% (Table 1). The new TDAM-only comparisons, without LMAA or JBR, improve actual coverage, recall, and AP.

Isolated TDAM results. Appendix E.1.3 changes only the selection supervision in Sparse-DETR. At ρ = 0.1, Table 8 reports:

BackboneSupervisionAPAP_s
ResNet-50DAM46.828.8
TDAM47.329.1
Swin-TDAM49.630.3
TDAM50.330.7

Isolated TDAM improves AP by 0.5 and 0.7 points on ResNet-50 and Swin-T, respectively, with AP_s gains of 0.3 and 0.4 points.

Actual update coverage and final detection recall. In Tables 9–10, coverage is the fraction of GT objects containing at least one center from the actual predicted update subset. Padding and out-of-image centers are excluded; overlapping boxes are assessed independently, and objects without an available center remain in the denominator.

At ρ = 0.1, small-object coverage increases from 98.1% to 99.9% with ResNet-50 and from 98.8% to 100.0% with Swin-T. Small-object AR increases from 47.7 to 48.6 and from 48.4 to 50.4. For Swin-T, medium- and large-object AR also improve, from 71.2 to 72.6 and from 85.3 to 87.3. AR follows COCO evaluation over IoU thresholds 0.50:0.05:0.95 with at most 100 detections per image. Swin-T small-object recall improves at all five update ratios:

Keeping ratio ρ0.10.20.30.40.5
DAM AR_s48.451.653.755.957.2
TDAM AR_s50.453.956.158.158.7
Improvement+2.0+2.3+2.4+2.2+1.5

At higher ratios, binary coverage reaches 100%. Appendix E.1.7 therefore also examines support per object through paired selection maps, changes in selected counts, feature-level distributions, normalized selection fractions, and coverage at higher minimum-support thresholds. The paired displays use confidence groups fixed by the DAM model and the same update ratio.

Swin-T and small-object AP. We suspect that Swin-T's local features provide limited foreground/background contrast for some very small objects. Swin uses shifted-window aggregation and hierarchical patch merging [1]; objects represented by few spatial tokens may retain enough information to improve coverage and recall while providing weaker cues for precise localization and confidence ranking, both of which affect AP. This could also affect the local geometry used by LMAA, making the complete model more sensitive than isolated TDAM to the backbone and update ratio.

Response to W2 and Q2: Repeated training and component interactions

Appendix E.1 reports mean and sample standard deviation across three training runs, using ResNet-50, full COCO, 50 epochs, and ρ = 0.1:

ComparisonFixed componentsControl APProposed AP
Independent / joint side predictionLMAA; original DAM48.4 ± 0.248.8 ± 0.2
Restricted random refill / TDAMLMAA; no JBR47.9 ± 0.548.9 ± 0.4
Without / with LMAAJBR and TDAM47.3 ± 0.249.0 ± 0.1

For boundary prediction, Table 4 retains the localization projection and compares independent and joint heads. AP_75 increases from 51.6 ± 0.5 to 52.4 ± 0.4. For supervision, Table 7 compares random refill and TDAM under the same protected-core constraint. TDAM improves mean AP by 1.0 point, and AP_75 increases from 52.3 ± 0.2 to 52.8 ± 0.3.

Effects without LMAA. The isolated DAM-versus-TDAM comparison in Table 8, reported above, changes only the supervision, with neither LMAA nor JBR. Tables 5–6 independently add JBR to Sparse-DETR without LMAA. At ρ = 0.1, JBR increases mean matched-box IoU from 76.1 to 78.6 with ResNet-50 and from 82.3 to 86.7 with Swin-T; normalized edge error decreases from 21.8 to 20.1 and from 19.1 to 15.4. Both localization measures are multiplied by 100.

Component interactions. Table 2 shows that JBR and TDAM gains are not additive. TDAM changes which features receive updates, while JBR learns to use boundary features for regression; these changes can interact during joint training. Table 11 repeats the comparison with both auxiliary components fixed: adding LMAA raises AP from 47.3 ± 0.2 to 49.0 ± 0.1 and AP_s from 29.2 ± 0.1 to 30.2 ± 0.3.

AuthorsOpenReview ↗

Runtime is profiled on a reserved GPU, and both suggested attention alternatives are evaluated.

Response to W3 and Q3: Runtime and the cost of using LMAA in six layers

Appendix E.3 reports repeated synchronized timing and an operation-level profile. On the reserved RTX 4090, with FP32, batch size one, and 800 × 1333 input, we use 50 warmup passes and six groups of 50 timed passes. Timing includes inference and GPU box postprocessing, excluding I/O and data transfer. Table 14 reports the following mean latencies:

ImplementationLatency (ms)
MSDA baseline26.9
LMAA53.8
LMAA with validity checking from the mask47.3

Computing candidate validity from the mask reduces LMAA latency by approximately 6.5 ms, or 12%.

Initial sampling with spatial derivatives and candidate-validity checking are the largest measured contributors: 8.3 and 8.4 ms of host interval, with 1152 and 1164 kernel launches per image. Together, they account for approximately 63% of the total host time and 66% of kernel launches reported in Table 17. Descriptor/Jacobian construction contributes 2.8 ms, metric and linear-system construction 1.4 ms, and the bounded position solve 3.3 ms of host interval. These fine-grained sampling and validation operations explain the runtime cost despite the small change in nominal FLOPs. Table 17 separately reports host intervals, which include dispatch and synchronization, and GPU kernel durations for the original LMAA implementation.

Using LMAA in more decoder layers repeats these sampling, derivative, validation, and correction operations. In the placement comparison (Table 15), the final layer gives 47.9 AP, compared with 47.3 for the final two layers and 46.9 for all six. We therefore use LMAA in the final layer, which achieves the best AP while performing the geometric correction only once.

Response to Q4: Alternative attention readers and stronger detector frameworks

Alternative attention readers. Table 3 and Appendix E.1.1 compare the attention designs using ResNet-50, full COCO, 50 epochs, ρ = 0.1, original DAM, and no JBR. Each alternative replaces the final decoder attention layer with the same final sampling-slot budget.

AttentionAPAP_75
Original MSDA46.850.4
Learned local correction45.650.2
Lite DETR-style key-aware attention46.251.4
LMAA47.952.0

The learned correction replaces the analytic position solve with a shared MLP receiving the local and query direction–magnitude descriptors, retaining the bounded update and content-energy weighting. The key-aware control uses learned query and sampled-key projections, d_K = 32, scaled dot products, and a softmax over sampling slots at the original positions. Equations (31)–(32) specify both controls. The content-only ablation instead applies an energy correction to the original query logits.

LMAA exceeds the learned correction and key-aware attention by 2.3 and 1.7 AP. Table 3 also compares the geometric ingredients: LMAA gives 47.9 AP, versus 47.0 for position correction alone, 46.9 for content weighting alone, and 46.8 for an isotropic update.

Transfer to four detector frameworks. Appendix E.5, Table 19, compares original attention and LMAA in DINO, D-FINE, Lite DETR, and RT-DETR. Each pair uses ResNet-50 and the same 12-epoch COCO schedule under the corresponding framework configuration:

FrameworkAP: original → LMAAAP_75AP_s
DINO49.3 → 50.653.8 → 55.832.5 → 33.8
D-FINE50.2 → 51.554.5 → 56.432.2 → 34.0
Lite DETR48.2 → 49.652.4 → 53.530.3 → 31.3
RT-DETR47.3 → 48.251.3 → 52.328.1 → 29.0

AP improves by 1.3, 1.3, 1.4, and 0.9 points, respectively, with AP_75 and AP_s improving in every pair. The D-FINE result shows that geometric feature reading remains useful alongside its localization refinements.

References

[1] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.

Reviewer_ZoxVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose Manifold-DETR, which retains the encoder sparsification framework of Sparse-DETR while modifying decoder cross-attention, bounding box refinement, and encoder token-selection supervision.

The method consists of three components: Local Manifold-Aligned Attention (LMAA), Joint Boundary Refinement (JBR), and a task-calibrated decoder attention map (TDAM). LMAA builds on feature-alignment principles related to methods such as PixLoc. It uses a direction-magnitude descriptor and local feature geometry to correct query-predicted sampling locations, then weights the sampled features based on their compatibility with the query. In the main experiments, LMAA is applied only to the final decoder layer. JBR combines boundary features with the query state to jointly refine the four box sides. TDAM uses matched-instance attention to supervise encoder token selection while preserving a high-demand core of the original selection target.

On COCO with ResNet-50 and 50 training epochs, Manifold-DETR improves over Sparse-DETR by 2.1 to 2.8 AP across encoder keeping ratios of 10% to 50%. It also achieves higher AP at a 10% keeping ratio than Sparse-DETR at 50%, although this reduction in updated tokens does not translate into lower inference latency.

Critical strengths

The idea of using local feature geometry to refine attention sampling locations is interesting. In particular, feature-based alignment offers a potentially useful way to correct the sampling offsets predicted by deformable attention and improve query-conditioned feature aggregation.

Critical weaknesses

1. The practical accuracy-latency trade-off is unclear.

Table 2 shows that adding LMAA increases inference latency from 27.18 ms to 58.67 ms, while AP improves from 46.8 to 47.9. This more than doubles latency despite a marginal increase in FLOPs. Although Appendix E.5 acknowledges this overhead, it does not provide a detailed breakdown of its causes. Local geometry computation and sampling-location correction may introduce operations that execute inefficiently on GPUs, but this remains unverified without profiling. Given the substantial latency increase and relatively modest accuracy gain, the practical benefit of the current implementation is unclear.

2. The distinction between JBR and existing boundary-based localization methods needs clarification.

The ablations show that JBR improves performance in the proposed framework, but they do not establish whether it offers advantages over existing approaches. JBR shares the broad idea of using boundary information for localization with BorderDet (ECCV 2020) and SABL (ECCV 2020). The paper would benefit from a clearer explanation of the differences and the advantages of its specific design.

3. The attention-utility mismatch is not directly validated.

The introduction argues that aggregate attention may allocate excessive encoder updates to already well-represented objects while giving too few updates to small or weakly represented objects. This is a central motivation for TDAM. However, the reported AP improvements do not directly establish that TDAM alleviates this imbalance. A more direct analysis connecting attention demand, actual token selection, and object-level detection performance is needed to support this explanation.

Questions for authors

1. Inference latency overhead of LMAA

Adding LMAA increases latency from 27.18 ms to 58.67 ms, while FLOPs increase only from 134.34G to 134.73G. Could the authors provide a latency breakdown to identify the main bottlenecks? In particular, how much overhead comes from Jacobian computation, construction of the local metric, and sampling-location correction? Is this overhead primarily a limitation of the current implementation, or is it inherent to the method?

2. Evidence for the attention-utility mismatch and the effectiveness of TDAM

Could the authors provide direct evidence that highly attended regions receive excessive encoder updates while small or weakly represented objects receive insufficient updates? How does TDAM change the actual selected-token distribution, object coverage, and detection recall for previously under-attended objects compared with the original DAM supervision?

3. Architecture illustration in Figure 1

The connection from encoder memory to the decoder's LMAA block is not clearly shown. Could the authors explicitly illustrate how LMAA reads keys and values from encoder memory? This would distinguish the cross-attention input from the proposal-based query initialization path.

4. Terminology at line 187

The term "slot p" is introduced without immediately explaining that it refers to a sampling point. Although the notation is clarified in Appendix A.1, using "sampling slot p" or "sampling-point index p" at its first occurrence would improve clarity.

5. Typo and readability in Figure 2

In the leftmost panel, "Reference Poitn" should be corrected to "Reference Point." The small labels are also difficult to read. A vector version with larger text would improve readability.

2 comments
AuthorsOpenReview ↗

Timing is repeated on an isolated GPU, LMAA latency drops 12%, and JBR is positioned against BorderDet/SABL.

Thank you for the helpful comments. We have added runtime profiling, boundary-refinement comparisons, and diagnostics of actual token allocation.

Response to W1 and Q1: Runtime measurements and implementation cost

The earlier timings were collected on a shared server. We repeated the measurements with the GPU reserved for benchmarking to reduce interference from concurrent workloads. Appendix E.3 reports one RTX 4090, FP32, batch size one, and 800 × 1333 inputs. Each configuration uses 50 warmup passes followed by six groups of 50 timed passes with GPU synchronization. We report the mean of the six group means, including inference and GPU box postprocessing and excluding file I/O and data transfer. Table 14 gives:

ImplementationLatency (ms)
MSDA baseline26.9
LMAA53.8
LMAA with validity checking from the mask47.3

The mask-based validity check reduces latency by approximately 6.5 ms, or 12%, while retaining the local descriptor, metric construction, and bounded sampling correction. Table 17 identifies the main costs of the original LMAA implementation:

LMAA stageHost interval (ms)GPU time (ms)Kernels/image
Initial sampling and spatial derivatives8.31.61152
Descriptor and Jacobian2.80.6408
Metric and linear system1.41.096
Bounded position solve3.30.5372
Candidate validity8.41.61164

Initial sampling and candidate validation account for approximately 63% of the total host time and 66% of kernel launches reported in Table 17. The many small sampling, derivative, indexing, and validity operations introduce dispatch and synchronization costs beyond their nominal FLOPs. Host intervals include dispatch and synchronization; GPU time sums the corresponding kernel durations. Local reads, derivatives, and solves add computational work, while the mask result shows that redundant implementation work can be reduced. The profile suggests further opportunities to consolidate these operations.

LMAA improves AP from 46.8 to 47.9, and the complete model reaches 48.9 AP. Section 5 now discusses these accuracy gains together with the additional inference latency, supported by the expanded analysis in Appendix E.3.

Response to W2: JBR in relation to BorderDet and SABL

Section 2.1 now discusses both methods. BorderDet uses BorderAlign to extract border features, and SABL localizes sides through bucket classification and regression. JBR combines multiscale boundary features with the final decoder query and coarse box to predict four joint, bounded side residuals.

Specifically, JBR samples all four sides using a localization-specific projection, then concatenates the boundary features, validity indicators, projected query, and coarse box. Each side prediction can therefore use information from the entire boundary and the object query. Section 3.4 gives the construction in Eqs. (10)–(12), with sampling and gradient details in Appendix C.2–C.3.

Table 4 fixes LMAA and original DAM supervision. A query-only residual head gives 47.9 AP, while boundary features with a shared projection give 48.1 AP. With the localization projection retained, independent and joint side prediction give the following means and sample standard deviations over three training runs:

PredictionAPAP_75
Independent sides48.4 ± 0.251.6 ± 0.5
Joint sides, JBR48.8 ± 0.252.4 ± 0.4

Tables 5–6 additionally compare Sparse-DETR with and without JBR using matched-box IoU, Mean Normalized Absolute Edge Error (MNAEE), and final recall. At ρ = 0.1, ResNet-50's mean IoU increases from 76.1 to 78.6 and MNAEE decreases from 21.8 to 20.1. With Swin-T, the corresponding changes are 82.3 to 86.7 and 19.1 to 15.4. IoU and MNAEE are multiplied by 100; MNAEE normalizes side errors by GT width and height.

AuthorsOpenReview ↗

TDAM's effect on update allocation is shown through per-object diagnostics, along with the Figure 1 and terminology clarifications.

Response to W3 and Q2: Allocation imbalance and the effect of TDAM

To examine the attention–utility mismatch, Appendix E.1.7 visualizes actual selected encoder positions. Figures 17–19 show how updates can concentrate on high-confidence objects while other annotated objects receive much less support. Feature levels and confidence groups are identified, and Figure 19 also reports selection density normalized by available candidate positions.

TDAM calibrates the selection target with normalized matched-instance attention and preserves its high-demand core. To isolate its effect, we train Sparse-DETR with DAM or TDAM at the same prescribed update ratio, with LMAA and JBR disabled. Tables 8–10 report AP, coverage by the actual predicted subset, and final COCO recall. At ρ = 0.1:

BackboneSupervisionAPAP_sCoverage_s (%)AR_s (%)
ResNet-50DAM46.828.898.147.7
TDAM47.329.199.948.6
Swin-TDAM49.630.398.848.4
TDAM50.330.7100.050.4

Coverage is the percentage of valid GT boxes containing at least one selected token center after excluding padding and out-of-image centers. AR follows the standard COCO evaluation. At the same encoder update ratio, TDAM increases small-object coverage by 1.8 and 1.2 percentage points and small-object AR by 0.9 and 2.0 points for ResNet-50 and Swin-T, respectively. The corresponding AP gains are 0.5 and 0.7 points. Tables 9–10 report higher small-object AR across all five update ratios for both backbones.

At larger budgets, binary coverage reaches 100% under both targets. Figures 20–45 examine the distribution of updates within covered objects through selected counts, feature-level distributions, selection fractions, and added or removed positions. These comparisons use MSDA+DAM and MSDA+TDAM, with confidence groups fixed by the DAM model.

For example, Figure 22 shows two low-confidence objects in image 74092 whose selected support increases from 5 to 39 and from 5 to 10 positions at ρ = 0.1. Figure 44 maps common, added, and removed positions. Figure 45 summarizes 79 objects across four paired examples using minimum-support thresholds from 1 to 32 positions, showing improved support for the low-confidence group at the tighter budget.

Response to Q3: The memory input to LMAA in Figure 1

The revised Figure 1 caption distinguishes encoder memory, which supplies the multiscale key and value fields, from encoder proposals, which initialize decoder queries and reference boxes. Section 3.1 explains how encoder updates are written back into the complete feature sequence for decoder sampling.

Specifically, LMAA projects the complete encoder memory into keys and values, using the stored spatial shapes and level indices to organize them as multiscale feature fields for each head. For each query, its reference box and predicted offsets specify the initial sampling positions. Bilinear interpolation reads the local keys and their spatial derivatives to compute a bounded correction of these positions. LMAA then resamples the keys at the corrected positions to adjust the attention logits through their compatibility with the query, while the existing deformable attention kernel reads values at the same positions. The weights are normalized across all levels and sampling points within each head; the weighted value sums are concatenated and passed through the output projection to update the query (Sections 3.2–3.3 and Appendix C.1).

Response to Q4 and Q5: Terminology and Figure 2 readability

At its first occurrence, p is now defined as the sampling point index. The revised Figure 2 caption also explains the reference point or box and sampling-slot identity. The label now reads Reference Point. We will enlarge the annotations and use a vector export checked at the final manuscript size. We have also included images at their original resolution in the supporting materials we submitted.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Chairs and Reviewers

We thank the three reviewers for the care and specificity they brought to this submission, and the Area Chair for coordinating a thorough and constructive discussion. The concerns raised are substantive, and the revision is stronger for them. Below we summarise the strengths identified, the concerns raised, and what the manuscript now contains.

1. Strengths the reviews identify

  • The motivation is well posed. L4iz notes that attention demand in Sparse-DETR need not indicate where extra encoder computation is most useful.
  • LMAA is the point of agreement. All three independently find it interesting to correct deformable-attention sampling offsets with local feature geometry, coupling coordinate refinement with content-dependent aggregation.
  • Gains are consistent and the ablations are substantial. AP improves over Sparse-DETR at every update ratio for both backbones, TDAM adds no inference-time branch, and the ablations cover all seven component combinations, both branches and LMAA placement.

2. Concerns raised, and what we have done

  • Accuracy–latency trade-off (all three). Latency rises from 27.18 to 58.67 ms while FLOPs barely move, without a breakdown. On a reserved RTX 4090, MSDA gives 26.9 ms, LMAA 53.8 ms, and 47.3 ms when candidate validity is read from the mask, 6.5 ms or 12% less. Table 17 profiles the stages: initial sampling with spatial derivatives 8.3 ms over 1152 launches and candidate validity 8.4 ms over 1164, about 63% of host time and 66% of kernel launches, against 2.8, 1.4 and 3.3 ms elsewhere. The cost comes from many small kernels and their dispatch and synchronisation rather than arithmetic; part is redundant. Section 5 now discusses both sides of the trade-off.
  • Direct evidence for the attention–utility mismatch (ZoxV, JkPr). AP alone cannot show that TDAM moves updates from well-represented to weakly represented objects, and Swin-T at 50 epochs has lower AP_S than Sparse-DETR between 20% and 50%. With LMAA and JBR disabled, TDAM alone improves AP by 0.5 and 0.7 and AP_S by 0.3 and 0.4, raises predicted-subset coverage 98.1% to 99.9% and 98.8% to 100.0%, and small-object AR 47.7 to 48.6 and 48.4 to 50.4; for Swin-T, AR_S improves at all five ratios. Appendix E.1.7 visualises the selected positions and their density, and we also report support per object. We attribute the Swin-T AP_S gap to shifted-window aggregation and patch merging, which leave very small objects with few tokens: enough for coverage and recall, weaker for localisation.
  • Uneven and non-additive gains, without variance (L4iz, JkPr). Three runs are reported with means and standard deviations: joint against independent sides 48.8±0.2 against 48.4±0.2, AP75 52.4±0.4 against 51.6±0.5; TDAM against constrained random refill 48.9±0.4 against 47.9±0.5; and with against without LMAA 49.0±0.1 against 47.3±0.2, AP_S 30.2±0.3 against 29.2±0.1. We agree LMAA is the principal contribution, a 1.7-point mean gain; JBR and TDAM are also measured alone. Table 2 explains the non-additivity through the interaction between which features TDAM sends updates to and how JBR uses boundary features for regression.
  • Narrow comparison scope (L4iz, JkPr). LMAA now replaces the decoder attention of DINO, D-FINE, Lite DETR and RT-DETR, improving AP by 1.3, 1.3, 1.4 and 0.9 points with AP75 and AP_S up in every pair. AI-TOD and DOTA are added: AI-TOD 19.2 to 21.6 AP, very-tiny 8.8 to 11.2; DOTA mean AP50 58.5 to 60.4, small vehicles 34.6 to 37.7. Among alternative readers, MSDA gives 46.8 AP, a learned correction 45.6, key-aware attention 46.2, LMAA 47.9. At 100, 300 and 900 queries, AP improves by 2.5, 2.1 and 1.6 points.
  • LMAA only helps in the final layer (L4iz, JkPr). AP is 46.8 without LMAA, 47.9 in the final layer, 47.3 in the final two, 46.9 in all six. Each extra layer repeats these stages, so the correction is applied once, after early queries and reference boxes are refined; Figure 16 traces 2741 slots in support.
  • JBR against BorderDet and SABL (ZoxV). Section 2.1 compares them: BorderDet extracts border features through BorderAlign, SABL localises sides through bucket classification and regression. JBR samples all four sides through a localisation projection and predicts four joint, bounded residuals from boundary features, validity indicators, the projected query and the coarse box (Eqs. 10 to 12, C.2 to C.3). Minor items: p is defined as the sampling point index, the Figure 1 caption separates encoder memory from proposals, and the Figure 2 label reads Reference Point.

We are grateful to all three reviewers and to the Area Chair for the time and expertise invested here; each concern has been answered with evidence, and we are explicit where a limitation remains. We would welcome a further look at the revised version before the discussion closes, and remain glad to clarify any point.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.