acceptodds
Under review as a conference paper at ICLR 2027

SeqDETR: Bridging the Query-Feature Gap in DETR via Deformable State Space Models

Abstract

Detection transformers refine object queries through a fixed three-stage pipeline: self-attention models relationships between queries, cross-attention aligns each query independently to image features, and feed-forward networks refine predictions locally. Despite their success, this pipeline has a structural gap, as no single component jointly models inter-query context and image features at the same time. Self-attention sees queries but not the image, while cross-attention sees the image but treats each query in isolation, remaining blind to what neighboring queries are attending to. We introduce SeqDETR, which fills this gap with a novel Deformable State Space Module (DSSM). For each object query, DSSM deformably samples K reference features from the encoder output. It then constructs an interleaved sequence of (image feature, query) pairs across all queries. A selective state space model processes this sequence and propagates a joint inter-query and image-feature representation through its recurrent state. This makes DSSM the first decoder operation in DETR-based models to simultaneously reason over both. Our analysis shows that the performance gain comes from two main factors: the interleaved sequence design that connects queries with their local image features, and the selective context propagation of the Mamba model. Component analysis further demonstrates that DSSM is highly efficient, as it approximates most of the full attention pipeline’s performance on its own. It is also complementary, yielding additional gains when combined with standard modules. SeqDETR achieves a 59.5% relative improvement on LVIS rare categories (+5.5 AP) and +4.0 AP on the 13,204-category V3Det benchmark. It also improves performance on COCO across different backbones and baselines. Code will be available upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Detection Transformers, Object Detection, Mamba, Query-Feature Interaction, Sequential Modeling.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jMM5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SeqDETR, a DETR-based object detection framework that addresses the limitations of conventional decoders in jointly modeling inter-query relationships and image features. It introduces a Deformable State Space Module (DSSM), which interleaves object queries with their sampled image features and uses a selective state space model to propagate contextual information efficiently. Experiments show consistent improvements across multiple benchmarks, particularly on LVIS rare categories and V3Det, demonstrating the effectiveness and complementarity of DSSM.

Critical strengths
  1. The proposed method improves several DETR-based baselines across COCO, LVIS, CrowdHuman, and V3Det, with particularly gains on rare categories and large-vocabulary detection.

  2. The paper evaluates different decoder component combinations, sequence designs, query orderings, and sequence modeling methods, providing useful evidence for the design choices.

Critical weaknesses
  1. Insufficiently established problem motivation. The paper argues that no standard decoder component jointly models inter-query context and image features, but it does not sufficiently demonstrate that this limitation is a significant bottleneck in existing DETR detectors. The authors should provide more direct evidence that this gap causes detection errors and cannot be adequately addressed by existing attention mechanisms.

  2. Limited novelty beyond deformable attention and state space modeling. DSSM uses deformable sampling to extract local image features for each query, a mechanism already employed by Deformable DETR. The main novelty therefore appears to lie in interleaving sampled features and queries and processing them with Mamba. The paper should more clearly distinguish this design from existing deformable attention and sequence modeling approaches, and demonstrate which component provides the essential benefit.

  3. Unclear efficiency advantage. Although DSSM is motivated by the linear complexity of state space models, it is added to the existing decoder rather than replacing the standard self-attention and cross-attention operations. The reported FLOPs also increase from 248.4G for the extra self-attention variant to 264.2G for the Mamba2 variant. Without comprehensive latency, memory, and throughput comparisons against the original baseline, the claim of efficiency remains insufficiently supported.

  4. Incomplete validation of the proposed mechanism. The ablations show that DSSM improves detection and that interleaved sequences and Mamba contribute to the gains. However, they do not fully establish that explicitly sharing sampled visual evidence across queries is the key reason for the improvement. More controlled comparisons with alternative feature-query fusion strategies would strengthen this claim.

  5. Incomplete efficiency and fairness evaluation. The paper compares several decoder variants, but the reported parameter counts and FLOPs alone are insufficient to assess practical efficiency. A fairer evaluation should compare the baseline and proposed method under consistent training settings and hardware, reporting inference latency, memory usage, and throughput alongside detection accuracy.

Questions for authors

See weakness.

Reviewer_9mFVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SeqDETR, a novel object detection method designed to address the insufficient interaction between object queries and image features in DETR-based detectors. To tackle this issue, the authors introduce a Deformable State Space Model (DSSM), which samples local image features around each query, interleaves the sampled features with query features to form a sequence, and employs the Mamba2 state space model to enhance information interaction between queries and image features. Experiments on COCO, LVIS, CrowdHuman, and V3Det demonstrate the effectiveness of SeqDETR across diverse object detection tasks.

Critical strengths

1.The paper identifies a concrete limitation in conventional DETR decoders and proposes a module that directly addresses the separation between query interaction and image-feature extraction. The motivation is intuitive, and the proposed module fits naturally into existing DETR architectures.

2.The method is evaluated with Deformable DETR, DINO, and MR-DETR, as well as different backbones. This provides initial evidence that DSSM is not restricted to a single detector configuration.

3.Experiments on COCO, LVIS, CrowdHuman, and V3Det cover standard detection, long-tailed recognition, crowded scenes, and large-vocabulary fine-grained detection.

Critical weaknesses

1.The novelty of the proposed ``query-feature isolation gap” appears overstated. The paper does not clearly establish that the identified limitation is fundamentally different from previously studied interactions between query reasoning and feature aggregation in DETR decoders. Consequently, it is unclear whether DSSM introduces a new modeling principle or mainly a new sequential implementation of joint query-feature processing.

2.DSSM introduces a strong dependence on an externally imposed query ordering. Although DETR object queries are inherently set-structured, SeqDETR converts them into a confidence-sorted causal sequence, and randomizing this order at inference decreases AP from 50.8 to 49.8. Therefore, the proposed joint modeling is not permutation invariant and relies on the encoder proposal confidence ranking as an additional structural prior.

3.The practical efficiency advantage is less clear than the linear-complexity discussion suggests. SeqDINO increases inference time from 87 ms to 110 ms and reduces throughput from 11.5 to 9.1 FPS. Moreover, while the method is theoretically linear in N(K+1), the paper does not empirically characterize how runtime and memory scale with the number of queries N and sampled features K.

Questions for authors

1.Since DSSM relies strongly on confidence-based query ordering, how robust is the method to errors or miscalibration in the encoder proposal scores used for sorting? The current experiments show that randomizing the order hurts performance, but do not clarify how sensitive DSSM is to partially incorrect confidence rankings.

2.SeqDINO improves AP but increases inference time from 87 ms to 110 ms. Since the paper emphasizes linear complexity in N(K+1), how do latency and memory empirically scale with the number of queries N and sampled features K?

3.The motivation for DSSM is that full attention over the joint query-feature sequence is prohibitively expensive. However, the experiments do not directly compare DSSM with self-attention applied to the same interleaved sequence. Under a reduced query or sampling setting where such a comparison is tractable, how does DSSM compare with full joint attention in accuracy and efficiency?

Reviewer_ytsGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SeqDETR, which adds a Deformable State Space Module (DSSM) to DETR decoders. DSSM interleaves object queries with deformably sampled image features and processes the sequence using Mamba2, with queries ordered by encoder confidence. Experiments cover three detector families, multiple backbones, and four benchmarks.

Critical strengths

1 The interleaved query-feature sequence provides a concrete way to combine local image evidence with communication across queries. The module is clearly formulated and fits into existing DETR decoders without changing their prediction objectives.

2 Improvements extend across Deformable DETR, DINO, and MR-DETR, using ResNet and Swin backbones. The gains therefore appear reasonably consistent across detector architectures rather than specific to one baseline.

3 This paper is easy to follow and understand.

Critical weaknesses

1 The “isolation gap” is overstated. Cross-attention already embeds image evidence into queries, which later self-attention can exchange. DSSM changes how this interaction occurs, but the paper does not establish a fundamental missing capability in standard DETR.

2 The method combines deformable sampling, token interleaving, and Mamba2. The experiments do not clearly isolate the benefit of inter-query recurrence from additional feature processing. Extra SA/CA layers receive different inputs and are therefore incomplete controls. A query-wise state-reset ablation is missing.

3 The COCO improvement comes at a noticeable runtime cost. SeqDINO gains 0.9 AP over DINO while increasing latency from 87 to 110 ms per image, about 26%. This limits the practical advantage of the proposed module despite its linear sequence complexity.

4 The claimed benefits for crowded scenes and small objects are not consistent across settings. Table 12 shows a smaller AP gain on dense scenes than on sparse scenes (+0.8 versus +0.9). With Swin-L, small-object AP decreases from 41.1 to 40.6 (Table 1). These results weaken the proposed explanation for where DSSM helps most.

Questions for authors

1 How does performance change when recurrent states are reset between query-feature groups, preserving local feature processing but removing inter-query propagation?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to AC and Reviewers

Dear Reviewers and Area Chair,

We sincerely thank all reviewers for their careful assessment and constructive feedback. We appreciate reviewer ytsG for recognizing the concrete interleaved query-feature formulation, its integration into existing decoders, and consistent improvements across architectures; reviewer 9mFV for highlighting the intuitive motivation, natural integration, and evaluation across detector families and diverse benchmarks; and reviewer jMM5 for recognizing the gains on rare-category and large-vocabulary detection and the systematic design ablations.

The revised manuscript incorporates the additional experiments discussed in our responses. Below, we summarize the main additions addressing concerns shared across the reviews.


1. Separating cross-query propagation from additional local processing.

To test whether the gain comes only from processing each query with its own sampled features, we train a control from scratch with recurrent states reset between query groups. This model achieves 50.3 AP, compared with 49.9 for DINO and 50.8 for full DSSM. Thus, local processing improves over DINO, while cross-query propagation provides a further 0.5 AP in this comparison.

We also add inference-only interventions using the same trained checkpoint. Removing other query tokens yields 50.7 AP, removing other queries’ sampled features yields 50.3, and restricting each query to its own group yields 50.2. These interventions suggest a larger contribution from shared sampled visual evidence than from other query tokens.


2. Comparing against full joint self-attention on the same token sequence.

We add joint-SA comparisons using the same interleaved query-feature inputs. At K=4/8/16, joint SA achieves 50.5/50.6/50.7 AP, versus 50.5/50.7/50.8 for DSSM. These results show that DSSM achieves accuracy comparable to full joint self-attention while offering faster inference at longer sequence lengths. Our contribution is the explicit joint query-feature formulation and its practical realization through DSSM, rather than an inherent accuracy advantage of Mamba2 over attention.


3. Reporting practical cost and the accuracy-latency trade-off.

We add end-to-end latency and peak allocated memory comparisons, together with scaling experiments over query count and sampled-feature count. At N=900, K=16, DSSM takes 110.5 ms/image, versus 307.7 for joint SA and 980.6 for joint LSTM: approximately 2.8× and 8.9× faster, respectively. The scaling results show that the latency advantage grows at longer sequences. However, DINO takes 87.28 ms/image, so DSSM adds approximately 27% latency. Although DSSM is faster than full joint attention and LSTM, further reducing its latency overhead relative to the baseline detector remains an important direction for future work.


4. Testing ordering sensitivity.

We add partial inference-time ranking perturbations: perturbing 10%, 30%, and 50% of ranks gives 50.7, 50.5, and 50.3 AP, respectively, versus 50.8 without perturbation and 49.8 under full randomization. These results show gradual degradation under the tested intervention. We distinguish this from training with random ordering, which achieves 50.2 AP, and spatial ordering, which achieves 50.5.


5. characterizing representation updates.

Finally, we add a quantitative representation analysis comparing DSSM with extra SA and CA. DSSM produces a larger increase in same-object coherence than extra SA (+7.1% vs. +5.0%) and a larger reduction in cross-object similarity (−11.7% vs. −5.1%), indicating more coherent representations among queries associated with the same object and stronger separation between different objects. DSSM also increases spatial-neighbor similarity by 11.1%, compared with 4.8% for extra SA. Together with the retrained state-reset control and information-removal experiments, these findings provide complementary evidence that DSSM’s joint query-feature processing supports cross-query contextual exchange and more discriminative query representations.


We thank the reviewers again for helping strengthen the evidence and clarify the scope of our claims. We hope these additions make the benefits, limitations, and contribution of SeqDETR easier to assess.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.