- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SeqDETR, a DETR-based object detection framework that addresses the limitations of conventional decoders in jointly modeling inter-query relationships and image features. It introduces a Deformable State Space Module (DSSM), which interleaves object queries with their sampled image features and uses a selective state space model to propagate contextual information efficiently. Experiments show consistent improvements across multiple benchmarks, particularly on LVIS rare categories and V3Det, demonstrating the effectiveness and complementarity of DSSM.
Critical strengths
-
The proposed method improves several DETR-based baselines across COCO, LVIS, CrowdHuman, and V3Det, with particularly gains on rare categories and large-vocabulary detection.
-
The paper evaluates different decoder component combinations, sequence designs, query orderings, and sequence modeling methods, providing useful evidence for the design choices.
Critical weaknesses
-
Insufficiently established problem motivation. The paper argues that no standard decoder component jointly models inter-query context and image features, but it does not sufficiently demonstrate that this limitation is a significant bottleneck in existing DETR detectors. The authors should provide more direct evidence that this gap causes detection errors and cannot be adequately addressed by existing attention mechanisms.
-
Limited novelty beyond deformable attention and state space modeling. DSSM uses deformable sampling to extract local image features for each query, a mechanism already employed by Deformable DETR. The main novelty therefore appears to lie in interleaving sampled features and queries and processing them with Mamba. The paper should more clearly distinguish this design from existing deformable attention and sequence modeling approaches, and demonstrate which component provides the essential benefit.
-
Unclear efficiency advantage. Although DSSM is motivated by the linear complexity of state space models, it is added to the existing decoder rather than replacing the standard self-attention and cross-attention operations. The reported FLOPs also increase from 248.4G for the extra self-attention variant to 264.2G for the Mamba2 variant. Without comprehensive latency, memory, and throughput comparisons against the original baseline, the claim of efficiency remains insufficiently supported.
-
Incomplete validation of the proposed mechanism. The ablations show that DSSM improves detection and that interleaved sequences and Mamba contribute to the gains. However, they do not fully establish that explicitly sharing sampled visual evidence across queries is the key reason for the improvement. More controlled comparisons with alternative feature-query fusion strategies would strengthen this claim.
-
Incomplete efficiency and fairness evaluation. The paper compares several decoder variants, but the reported parameter counts and FLOPs alone are insufficient to assess practical efficiency. A fairer evaluation should compare the baseline and proposed method under consistent training settings and hardware, reporting inference latency, memory usage, and throughput alongside detection accuracy.
Questions for authors
See weakness.
General Response to AC and Reviewers
Dear Reviewers and Area Chair,
We sincerely thank all reviewers for their careful assessment and constructive feedback. We appreciate reviewer ytsG for recognizing the concrete interleaved query-feature formulation, its integration into existing decoders, and consistent improvements across architectures; reviewer 9mFV for highlighting the intuitive motivation, natural integration, and evaluation across detector families and diverse benchmarks; and reviewer jMM5 for recognizing the gains on rare-category and large-vocabulary detection and the systematic design ablations.
The revised manuscript incorporates the additional experiments discussed in our responses. Below, we summarize the main additions addressing concerns shared across the reviews.
1. Separating cross-query propagation from additional local processing.
To test whether the gain comes only from processing each query with its own sampled features, we train a control from scratch with recurrent states reset between query groups. This model achieves 50.3 AP, compared with 49.9 for DINO and 50.8 for full DSSM. Thus, local processing improves over DINO, while cross-query propagation provides a further 0.5 AP in this comparison.
We also add inference-only interventions using the same trained checkpoint. Removing other query tokens yields 50.7 AP, removing other queries’ sampled features yields 50.3, and restricting each query to its own group yields 50.2. These interventions suggest a larger contribution from shared sampled visual evidence than from other query tokens.
2. Comparing against full joint self-attention on the same token sequence.
We add joint-SA comparisons using the same interleaved query-feature inputs. At K=4/8/16, joint SA achieves 50.5/50.6/50.7 AP, versus 50.5/50.7/50.8 for DSSM. These results show that DSSM achieves accuracy comparable to full joint self-attention while offering faster inference at longer sequence lengths. Our contribution is the explicit joint query-feature formulation and its practical realization through DSSM, rather than an inherent accuracy advantage of Mamba2 over attention.
3. Reporting practical cost and the accuracy-latency trade-off.
We add end-to-end latency and peak allocated memory comparisons, together with scaling experiments over query count and sampled-feature count. At N=900, K=16, DSSM takes 110.5 ms/image, versus 307.7 for joint SA and 980.6 for joint LSTM: approximately 2.8× and 8.9× faster, respectively. The scaling results show that the latency advantage grows at longer sequences. However, DINO takes 87.28 ms/image, so DSSM adds approximately 27% latency. Although DSSM is faster than full joint attention and LSTM, further reducing its latency overhead relative to the baseline detector remains an important direction for future work.
4. Testing ordering sensitivity.
We add partial inference-time ranking perturbations: perturbing 10%, 30%, and 50% of ranks gives 50.7, 50.5, and 50.3 AP, respectively, versus 50.8 without perturbation and 49.8 under full randomization. These results show gradual degradation under the tested intervention. We distinguish this from training with random ordering, which achieves 50.2 AP, and spatial ordering, which achieves 50.5.
5. characterizing representation updates.
Finally, we add a quantitative representation analysis comparing DSSM with extra SA and CA. DSSM produces a larger increase in same-object coherence than extra SA (+7.1% vs. +5.0%) and a larger reduction in cross-object similarity (−11.7% vs. −5.1%), indicating more coherent representations among queries associated with the same object and stronger separation between different objects. DSSM also increases spatial-neighbor similarity by 11.1%, compared with 4.8% for extra SA. Together with the retrained state-reset control and information-removal experiments, these findings provide complementary evidence that DSSM’s joint query-feature processing supports cross-query contextual exchange and more discriminative query representations.
We thank the reviewers again for helping strengthen the evidence and clarify the scope of our claims. We hope these additions make the benefits, limitations, and contribution of SeqDETR easier to assess.