acceptodds
Under review as a conference paper at ICLR 2027

Motion-Aware Transformer for Multi-Object Tracking

Abstract

Multi-object tracking with query propagation maintains object identities by carrying track queries across frames, but the propagated state can become misaligned with the current observation before it interacts with newly initialized detection queries. We show that this temporal state misalignment can create query ownership conflicts, where a detection query represents an existing target better than its designated track-query owner. To address this problem, we introduce MATR, an end-to-end tracking framework built around a Motion-Aware Transformer (MAT) module. MAT uses current-frame evidence to pre-align both the feature and spatial states of inherited track queries before joint decoding. We further derive a geometric ambiguity measure and show empirically that ownership-conflict risk increases sharply with ambiguity, while MAT applies larger corrections to more difficult states. Across DanceTrack, SportsMOT, and BDD100K, MATR consistently improves association while introducing only modest computational overhead. On DanceTrack, MATR reaches 71.3 HOTA with 43M parameters and 179 GFLOPs. On BDD100K, without additional tracking data or external detector proposals, MATR reaches 54.7 mTETA and 41.6 mHOTA, the strongest result we are aware of among query-propagation trackers trained under this native-data setting. Source code will be publicly released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-object tracking, End-to-end tracking, Query propagation, Transformer, Motion modeling, Data association, Temporal alignment, Track queries

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_n8KEOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors identify temporal state misalignment as a potential issue in query-propagation-based multi-object tracking methods such as MOTR, where track queries inherited from previous frames may no longer accurately represent targets in the current frame, particularly under significant object or camera motion.

To address this issue, the authors propose a lightweight pre-alignment module, termed the Motion-Aware Transformer (MAT), which leverages current-frame observations to update both the feature representations and spatial references of inherited track queries before joint decoding. Built upon MOTR, the proposed MATR framework achieves substantial improvements in tracking performance.

Critical strengths

The authors provide a reasonable analysis of the limitations of track-query propagation in MOTR-style trackers. The proposed pre-alignment module is also intuitive and conceptually simple, offering a straightforward way to mitigate temporal state misalignment.

Critical weaknesses
  1. The authors use different evaluation splits across their ablation studies, which is inconsistent with standard experimental practice and makes the results difficult to compare systematically. The explanation provided in Lines 361–362 is not sufficiently convincing, as the absence of a reported MOTR validation result does not prevent the authors from evaluating the MOTR checkpoint on the validation set. The authors should adopt a consistent evaluation protocol across their ablation studies to ensure clearer and more rigorous comparisons.

  2. Table 3(b) lacks a corresponding baseline without MAT (i.e., Bl-IMP evaluated on the same validation split). Without this baseline, it is difficult to assess the individual contributions of feature and position alignment relative to the original configuration, or even determine whether each component provides a positive improvement.

  3. The effectiveness of spatial alignment is not sufficiently demonstrated in Table 3(b). Specifically, combining feature and position alignment improves HOTA by only 0.2 points compared with feature alignment alone (58.1 vs. 57.9). Such a marginal improvement could potentially result from random fluctuations during end-to-end MOT training. The authors should provide additional ablation results to establish the robustness of this improvement. Furthermore, it is unclear whether the auxiliary L1 alignment loss (Eq. (5)) is retained when position alignment is disabled in the Feature-only configuration. This implementation detail should be clarified.

Questions for authors

In the Weaknesses section, we have raised three key concerns and questions that require direct, point-by-point responses from the authors. We consider these to be substantive, fact-based issues concerning the paper's methodology and experimental evidence, rather than matters of subjective interpretation or preference. If the authors satisfactorily address these concerns, and no other reviewers raise additional issues that I consider fundamental or potentially fatal to the paper, I would be willing to raise my score.

Beyond the issues discussed above, we would also like the authors to clarify how MAT handles inherited track queries whose corresponding targets are absent from the current frame. Specifically, are the pre-aligned states retained or discarded, and can they affect the states propagated to subsequent frames? Does the handling of such queries differ between training and inference?

We also encourage the authors to improve the visual presentation of Fig. 2 to make it more polished and publication-ready. Specifically, some modules and graphical elements appear misaligned, and the visual balance between different shapes (e.g., squares and triangles) could be improved. More consistent alignment, spacing, and visual centering would enhance the overall clarity and professionalism of the figure.

1 comment

Dear Reviewer,

Thank you again for your valuable feedback. We have provided a detailed response to your concerns and would greatly appreciate it if you could take a look when you have a chance.

We would be grateful to know whether our clarifications have adequately addressed your main concerns, or if there are any remaining questions that we could further clarify during the discussion period.

Thank you for your time and consideration.

Reviewer_zDZeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MATR addresses temporal misalignment of inherited track queries in MOTR-style multi-object tracking. It places a one-layer, current-frame-conditioned MAT module before the original joint decoder. MAT updates inherited track-query features and predicts an aligned box and spatial reference under an auxiliary L1 loss; detection queries and target-consistent identity assignment retain their original roles in joint decoding. The paper defines an ownership margin and a geometric ambiguity measure, and separates MAT from modifications to its strengthened MOTR baseline. The complete system reports competitive results across DanceTrack test (71.3 HOTA), SportsMOT test (72.7 HOTA), and BDD100K validation (54.7 mTETA), with modest incremental inference cost for MAT.

Critical strengths
  1. A substantial, association-centered gain over the evaluated strengthened baseline. The paper reports its baseline modifications separately. In the five-epoch DanceTrack test ablation, adding MAT to Bl-IMP raises HOTA from 58.8 to 63.6, AssA from 45.2 to 52.7, and IDF1 from 59.2 to 66.4. The analysis also shows that MAT reduces intermediate track-state center error across all 25 DanceTrack validation videos examined. These results provide meaningful evidence that MAT is effective in the evaluated setting.

  2. Competitive tracking results at modest incremental inference cost. MATR-R50 reports 42M parameters, 175 GFLOPs, and 18 FPS, compared with 40M, 173 GFLOPs, and 19 FPS for the listed MOTR configuration. The complete tracker reports 71.3 HOTA on DanceTrack test, 72.7 HOTA on SportsMOT test, and 54.7 mTETA on BDD100K validation without external detector proposals. These are useful results for a query-propagation system evaluated across different tracking settings.

Critical weaknesses

The paper has not shown that a consequential track-query error remains after MOTR's six joint decoder layers under the final training schedule. A query inherited from the previous frame naturally begins with a stale spatial reference. In MOTR, however, inherited track queries pass through six joint decoder layers that read current-frame features, predict boxes, and supervise track-query boxes at every layer. Figure 1 compares directly propagated and MAT-aligned track states, but does not report how track-query localization changes through the six original layers or what error remains after the last layer. The Bl-IMP versus Bl-IMP+MAT comparison shows a substantial gain after five training epochs, whereas the final DanceTrack models train for 20 epochs without a paired Bl-IMP control. Further training may allow the original decoder to narrow this gap. Thus, the results establish the benefit of MAT in the short-schedule experiment, but not yet the persistence of the proposed residual problem in the final setting.

The controlled five-epoch gain from Bl-IMP to Bl-IMP+MAT is substantial, but the final DanceTrack models train for 20 epochs without a paired 20-epoch Bl-IMP result. The baseline may be insufficiently trained in the short ablation and could narrow the gap as its existing layers learn to refine inherited queries. If a consequential residual error persists after sufficient training, one plausible explanation is that track queries interact with detection queries before their states have aligned to the current frame. The paper has not tested this explanation directly. A six-layer control with asymmetric self-attention only in the first layer—detection queries may attend to both types, while inherited track queries attend only to track queries—would retain current-frame cross-attention for both types and ordinary joint attention in layers 2–6. It would test whether delaying detection-to-track interaction helps without adding a layer; an ordinary extra joint layer could further check the effect of depth.

This distinction also matters relative to MOTRv3's RFS and CO-MOT's COLA, which alter the early supervision shaping detect–track interaction, and MeMOTR's reorganized first decoder layer. These approaches differ from MAT, but the paper does not establish whether they leave the same post-refinement track-query problem unresolved. The present evidence supports MAT's benefit under the tested short schedule; its specific mechanism and incremental contribution remain less certain.

Questions for authors
  1. Under the final 20-epoch DanceTrack training schedule, how do Bl-IMP and Bl-IMP+MAT compare with all other settings held fixed? For existing targets, can the authors report track-query box quality after each joint decoder layer, particularly after layer six, together with track confidence and final association outcomes? Which layer's predictions are used to calculate the ownership margin?

  2. Can the authors test the early-interaction hypothesis in a six-layer model using an asymmetric self-attention mask only in layer one: detection queries attend to both query types, inherited track queries attend only to track queries, and both types cross-attend to current-frame encoder features? Keep the original supervision and use ordinary joint attention in layers 2–6. How does this affect layer-wise track-query refinement and final AssA relative to Bl-IMP and MAT under the same training schedule? Please report actual parameters and FLOPs. An ordinary additional joint layer would provide a further depth control.

  3. Do MOTRv3's RFS, CO-MOT's COLA, or MeMOTR's first-layer arrangement improve the same residual track-query problem after joint decoding? A representative controlled diagnostic, or a precise explanation of how these mechanisms differ on that specific failure, would clarify MAT's contribution.

I would be happy to raise my rating if additional experiments convincingly show that the proposed method addresses query conflicts more effectively than existing approaches and provide further insight into why it works.

6 comments

Dear Reviewer,

Thank you again for your valuable feedback. We have provided a detailed response to your concerns and would greatly appreciate it if you could take a look when you have a chance.

We would be grateful to know whether our clarifications have adequately addressed your main concerns, or if there are any remaining questions that we could further clarify during the discussion period.

Thank you for your time and consideration.

Dear Reviewer,

Thank you again for your valuable feedback. We have provided a detailed response to your concerns and would greatly appreciate it if you could take a look when you have a chance.

We would be grateful to know whether our clarifications have adequately addressed your main concerns, or if there are any remaining questions that we could further clarify during the discussion period.

Thank you for your time and consideration.

Dear Reviewer,

Thank you again for your valuable feedback. We have provided a detailed response to your concerns and would greatly appreciate it if you could take a look when you have a chance.

We would be grateful to know whether our clarifications have adequately addressed your main concerns, or if there are any remaining questions that we could further clarify during the discussion period.

Thank you for your time and consideration.

Reviewer_yXjSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies temporal state misalignment as a problem in query-propagation-based multi-object tracking: a propagated track query may remain the legal owner of an object while its feature and spatial reference become stale in the current frame. The authors propose MATR, which uses a Motion-Aware Transformer (MAT) to pre-align both the feature and spatial states of inherited track queries using current-frame encoder features before joint decoding with detection queries. The paper further introduces an ownership margin and a geometric ambiguity ratio to analyze query ownership conflicts. Experiments on DanceTrack, SportsMOT, and BDD100K show consistent improvements, particularly in association metrics such as AssA, IDF1, and mAssocA, with modest computational overhead. Ablations support the contributions of feature alignment, spatial alignment, and auxiliary L1 supervision.

Critical strengths
  1. Clear problem formulation and targeted intervention. The paper isolates a concrete failure mode of query-propagation trackers: the propagated query can preserve identity while becoming temporally misaligned with the current observation. This provides a useful conceptual distinction between identity ownership and state alignment.

  2. Consistent empirical evidence with low computational overhead. MAT improves association performance across three benchmarks, and the ablations show that jointly updating feature and spatial states is more effective than updating either component alone. The method introduces only modest increases in parameters and computation relative to MOTR.

Critical weaknesses
  1. The improvement over the original MOTR is not solely attributable to MAT. The strengthened baseline already improves DanceTrack HOTA from 54.2 to 58.8, while adding MAT further increases it to 63.6 in the ablation setting. Although the authors explicitly acknowledge this, the main presentation could more clearly separate the contribution of baseline improvements from that of MAT.

  2. The mechanism analysis is concentrated mainly on DanceTrack. The paper demonstrates performance gains on SportsMOT and BDD100K, but the ambiguity and ownership-conflict analysis is not comparably validated on those datasets. Additional cross-dataset analysis would strengthen the claim that the proposed mechanism generalizes broadly.

Questions for authors

Can the authors report ownership-conflict rates, ambiguity-ratio distributions, and pre/post-MAT alignment errors on SportsMOT and BDD100K, in addition to the existing DanceTrack analysis?

How sensitive are the results to the auxiliary alignment-loss weight, the inherited-track IoU filtering threshold, and the track-query dropping probability?

Comments on the paper

2 comments

Thank you for your follow-up. We appreciate the opportunity to further clarify the underlying mechanism.

1. Why can MAT correct the inherited track state more effectively than MOTR's six-layer decoder?

Because MOTR attempts to correct inherited track states after they have entered joint decoding, where competition with detection queries has ALREADY BEGUN, whereas MAT mitigates their misalignment before this competition.

The key distinction is that the two query types enter joint decoding with different temporal states. Detection queries operate on current-frame observations, whereas inherited track queries may carry outdated features and spatial references. When these queries interact without prior alignment, a detection query may represent an existing target better than its designated track-query owner, creating an ownership conflict. MAT therefore does not need to modify detection queries: by aligning inherited track queries with current-frame observations, it mitigates the state mismatch underlying these conflicts.

Although target-consistent assignment reserves each existing target for its inherited track query, it does not guarantee the best representation. Consequently, the joint decoder must refine a misaligned track query while interacting with competing detection queries. Additional decoder refinement does not necessarily resolve this conflict, as demonstrated by our layer-wise experiments. For its formal definition and illustration, please refer to Section 3.1 (Eq. (1)) and Fig. 1(b).

MAT addresses the problem earlier. It updates inherited track features using current-frame observations, predicts a corrected spatial reference, and directly supervises alignment through an auxiliary L1 loss. The corrected reference also guides subsequent deformable attention toward the current target.

Thus, MAT does not outperform six decoder layers simply through stronger refinement. It changes the state entering joint decoding, mitigating ownership conflicts before they develop.

This also explains the asymmetric-attention result. Restricting track-to-detection attention changes the interaction pattern but does not explicitly correct inherited spatial references or provide alignment supervision. It therefore cannot reproduce MAT's benefit.

Fig. 3 shows that inherited spatial ambiguity strongly predicts ownership conflicts, while MAT reduces this ambiguity before joint decoding. Furthermore, Fig. 8 shows that MATR reduces the residual ownership-conflict rate from 25.29% to 9.63%, measured after all six joint decoder layers. Thus, pre-alignment provides benefits that persist through subsequent refinement.

2. Does the improvement accumulate through recurrent propagation?

Yes. Since refined track states propagate to subsequent frames, corrections from earlier frames can improve later initialization.

However, MAT also provides measurable correction within the current frame. As reported previously, the same inherited track states improve from 55.3 IoU before MAT to 57.4 IoU after MAT, before joint decoding. Fig. 3(b) further confirms this immediate correction, showing a 34.9% reduction in normalized center error across paired pre/post-MAT states.

Therefore, MATR's higher initial IoU is consistent with its recurrent effect: MAT corrects the current state, which is then propagated to benefit subsequent frames.

Importantly, the controlled Bl-IMP comparison evaluates the overall contribution of this mechanism under identical training settings. After 20 epochs, MATR maintains gains of 3.6 HOTA, 3.8 AssA, and 5.4 IDF1, confirming its advantage under the final training schedule.

Together, these results support a consistent explanation: MAT corrects temporal misalignment before query interaction, reduces residual ownership conflicts after joint decoding, and propagates improved states across frames. These are complementary aspects of the same mechanism rather than competing explanations.

We hope this clarification resolves the remaining concerns regarding MAT's mechanism and contribution. We would appreciate your reconsideration of the rating.

Dear Reviewer,

Thank you again for your valuable feedback. We have provided a detailed response to your concerns and would greatly appreciate it if you could take a look when you have a chance.

We would be grateful to know whether our clarifications have adequately addressed your main concerns, or if there are any remaining questions that we could further clarify during the discussion period.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.