- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Manifold-DETR, an object detection framework built on Sparse-DETR to improve detection accuracy under limited encoder token updates. It introduces three components, first Local Manifold-Aligned Attention (LMAA): Uses local feature geometry to correct deformable attention sampling locations and improve feature aggregation, second Joint Boundary Refinement (JBR): Samples features along predicted bounding box boundaries to jointly refine all four sides and last Task-Calibrated Decoder Attention Map (TDAM): Uses instance-level attention from matched object queries to improve encoder token selection during training. Experiments on COCO with ResNet-50 and Swin-T show consistent improvements over Sparse-DETR across different encoder keeping ratios. Ablation studies evaluate the contributions of each component.
Critical strengths
- Clear motivation: The paper identifies an important limitation of Sparse-DETR: attention demand does not necessarily reflect where additional encoder computation is most useful.
- Interesting attention improvement: LMAA addresses a limitation of deformable attention by using local feature geometry to refine query-predicted sampling locations rather than relying entirely on predicted offsets.
- No inference overhead from TDAM: The proposed task-calibrated supervision improves token selection without adding an additional inference-time branch.
- Good ablation studies: The experiments reasonably isolate the contributions of the proposed components and demonstrate consistent improvements over Sparse-DETR across different token keeping ratios.
Critical weaknesses
- Outdated baseline and limited comparisons: The framework is built on Sparse-DETR, an older detector. Although Table 1 is presented as a comparison with state-of-the-art detectors, it lacks stronger recent architectures such as DINO, RT-DETR, and D-FINE. This limits the evaluation of the method's competitiveness and generalizability.
- Limited dataset evaluation: Experiments are conducted only on COCO. Evaluation on additional datasets, particularly those with small or difficult objects, would better support the proposed motivation.
- Significant inference overhead: Although Manifold-DETR improves accuracy with fewer updated encoder tokens, its inference latency increases substantially. At a 10% keeping ratio, latency increases from 27.18 ms to 62.97 ms compared with Sparse-DETR, despite a negligible increase in FLOPs. This weakens the efficiency claims.
- Unequal contribution of components: LMAA provides most of the improvement, while JBR and TDAM offer relatively small additional gains. Removing JBR reduces AP by only 0.2, while removing TDAM reduces it by 0.4.
- Limited effectiveness across decoder layers: LMAA performs best when applied only to the final decoder layer. Applying it to more layers reduces accuracy, raising questions about its limited effectiveness when extended to earlier decoder layers.
Questions for authors
- Q1 Can LMAA be integrated into newer architectures such as DINO, RT-DETR, and D-FINE ?
- Q2 Can the authors evaluate the method on additional datasets, particularly those containing small, difficult, or weakly attended objects?
- Q3 How do the proposed components behave with different numbers of object queries, such as 100, 300, and 900?
- Q4 Why does inference latency increase substantially despite the negligible increase in FLOPs? Which operations introduce the largest overhead?
- Q5 Why does applying LMAA to additional decoder layers reduce performance?
2 comments
LMAA is the main contributor, the final decoder layer is the best placement, and gains hold at 100–900 queries.
Response to W4: The respective contributions of LMAA, JBR, and TDAM
LMAA is the principal contribution. JBR improves boundary localization, and TDAM calibrates encoder update supervision. The revision adds repeated comparisons and isolated measurements of these roles.
Table 11 repeats the comparison with JBR and TDAM fixed. Across three training runs, JBR+TDAM reaches 47.3 ± 0.2 AP, and the complete model reaches 49.0 ± 0.1 AP, a 1.7-point mean improvement from LMAA. AP_75 changes from 51.5 ± 0.3 to 52.7 ± 0.0. Values are means and sample standard deviations.
The design comparisons have also been repeated. Table 4 fixes LMAA and original DAM: independent side prediction gives 48.4 ± 0.2 AP and joint prediction gives 48.8 ± 0.2. Table 7 fixes LMAA and disables JBR: restricted random refill gives 47.9 ± 0.5 AP and TDAM gives 48.9 ± 0.4.
We also evaluate each auxiliary component without LMAA. For JBR, Tables 5–6 compare Sparse-DETR with and without boundary refinement. At ρ = 0.1, ResNet-50's mean matched-box IoU increases from 76.1 to 78.6 and Mean Normalized Absolute Edge Error decreases from 21.8 to 20.1. Swin-T gives corresponding changes of 82.3 → 86.7 and 19.1 → 15.4. Both measures are multiplied by 100; the edge error normalizes each side by GT width or height.
For TDAM, Tables 8–10 change only DAM supervision. At the same update ratio, small-object coverage by actual selected centers increases from 98.1% to 99.9% with ResNet-50 and from 98.8% to 100.0% with Swin-T. Small-object AR improves from 47.7 to 48.6 and from 48.4 to 50.4; isolated AP also improves from 46.8 to 47.3 and from 49.6 to 50.3. The paired visualizations in Appendix E.1.7 show the corresponding changes in the selected subsets.
Response to W5 and Q5: Why the final decoder layer is the best placement
Table 15 compares placements using ResNet-50, full COCO, 50 epochs, and ρ = 0.1. AP is 46.8 without LMAA, 47.9 with LMAA in the final layer, 47.3 in the final two layers, and 46.9 in all six layers. Our most plausible explanation is the interaction between early-query reliability and the local range of geometric correction.
LMAA aligns local features with the current query descriptor. For the i-th sampling point, Jᵢ is the descriptor Jacobian at the current iterate (initialized at xᵢ⁰), rᵢ is the residual between the current query descriptor and the target descriptor t, and λ is the damping weight:
vᵢ = −(JᵢᵀJᵢ + λI)⁻¹ Jᵢᵀrᵢ, rᵢ = zᵧ(k(xᵢ⁰)) − zᵧ(t)
The damped update is capped at 0.5 feature cell. Early queries and reference boxes may still be ambiguous, so improving a local match can reinforce an intermediate interpretation and affect subsequent query and box refinement. Repeating this correction across layers may compound that bias, since the target descriptor and sampling state change at each layer. Applying LMAA at the final layer lets local alignment operate after the preceding stages have refined the object representation and spatial reference.
Figure 16 illustrates this local behavior using 2741 sampling slots initially outside their matched boxes in four final-layer examples. For initial distances of 0–0.1 GT-box diagonals, 39.4–51.7% of slots enter the box; for distances of 0.1–0.5, the fraction is 17.1–35.0%. This pattern is consistent with our interpretation that bounded geometric correction is most useful once the sampling neighborhood is already close to the object.
Response to Q3: Different numbers of object queries
Appendix E.1.5, Table 12, evaluates 100, 300, and 900 queries. Each model is trained for 50 epochs on full COCO with ResNet-50 and ρ = 0.1, using the corresponding query count.
| Number of queries | 100 | 300 | 900 |
|---|---|---|---|
| Sparse-DETR AP | 45.9 | 46.8 | 48.2 |
| Manifold-DETR AP | 48.4 | 48.9 | 49.8 |
| AP improvement | +2.5 | +2.1 | +1.6 |
| Sparse-DETR AP_s | 26.5 | 28.8 | 30.4 |
| Manifold-DETR AP_s | 28.2 | 29.9 | 31.5 |
Manifold-DETR improves AP and AP_s at all three query budgets. Absolute AP increases with more queries, while the AP gain over the matched baseline decreases from 2.5 to 1.6 points.
LMAA improves four modern detectors and two aerial datasets, with measured inference overhead.
Thank you for the careful review. We have added experiments on four detection frameworks and two aerial datasets, repeated the key component comparisons, and expanded the query-count, timing, and feature-reading analyses.
Response to W1 and Q1: LMAA in modern detection frameworks
LMAA improves DINO, D-FINE, Lite DETR, and RT-DETR. Appendix E.5, Table 19, compares each framework's original decoder attention with its LMAA adaptation. Each pair uses ResNet-50 and the same 12-epoch COCO schedule under the corresponding framework configuration.
The AP improvements are 1.3, 1.3, 1.4, and 0.9 points, with gains in AP_75 and AP_s in all four pairs. D-FINE is particularly informative: its stronger localization design still benefits from improved feature reading, reaching 51.5 AP and 56.4 AP_75. Lite DETR tests another attention architecture, while DINO and RT-DETR extend the comparison to different detector configurations.
LMAA operates on a query, a spatial reference, and multiscale memory. This interface allows its geometric correction and content aggregation to be integrated independently of the Sparse-DETR selector and TDAM supervision.
Response to W2 and Q2: Additional datasets with small and difficult objects
Appendix E.6 adds AI-TOD and DOTA, comparing Sparse-DETR with the complete Manifold-DETR. Both use ResNet-50 and ρ = 0.1, with the same encoder update budget.
For AI-TOD, the models use 72,000 training updates, 11,214 training images, and 2,804 validation images. Evaluation follows the dataset's four size categories and allows at most 1,500 detections per image. Table 20 reports AP over IoU thresholds 0.50:0.05:0.95:
Overall AP improves by 2.4 points, including a 2.4-point gain for very tiny objects. Table 21 reports AR for the very tiny, tiny, small, and medium groups: 14.6/33.9/36.5/38.6 for Sparse-DETR and 17.2/35.2/38.9/40.1 for Manifold-DETR.
For DOTA, Table 22 reports horizontal-box AP_50 on image tiles after 24 epochs, averaged over all 16 categories. Mean AP_50 increases from 58.5 to 60.4, and small-vehicle AP_50 improves from 34.6 to 37.7. The complete category results are provided in the table.
Response to W3 and Q4: Inference overhead despite a small FLOP increase
We repeated timing with the benchmarking GPU reserved to avoid interference from concurrent jobs on the shared server. Appendix E.3 specifies one RTX 4090, FP32, batch size one, and 800 × 1333 inputs. Timing covers inference and GPU box postprocessing, excluding I/O and data transfer. Each configuration receives 50 warmup passes and six groups of 50 timed passes with GPU synchronization. Table 14 reports the following mean latencies:
Computing candidate validity from the mask reduces LMAA latency by approximately 6.5 ms, or 12%, while retaining the local geometric reading procedure.
Table 17 identifies the main bottlenecks of the original LMAA implementation. Initial sampling and spatial derivatives account for 8.3 ms of host interval and 1152 kernel launches per image; candidate validity accounts for 8.4 ms and 1164 launches. Together, these account for approximately 63% of the total host time and 66% of kernel launches reported in Table 17. Descriptor/Jacobian computation takes 2.8 ms of host interval, metric and linear-system construction 1.4 ms, and the bounded position solve 3.3 ms.
Local sampling, indexing, validity processing, and many small kernel launches explain the runtime overhead beyond the arithmetic counted by FLOPs. The mask implementation directly reduces one of the largest stages.