acceptodds
Under review as a conference paper at ICLR 2027

Reliable Identity Memory for RGB-T Multi-Object Tracking

Abstract

RGB-thermal multi-object tracking aims to localize multiple targets and preserve their identities across paired visible and thermal videos. Despite effectively fusing multimodal information, existing methods struggle to maintain consistent identity associations over time. This challenge is particularly acute for tiny objects, whose weak spatial evidence often leads to intermittent detections, disrupting temporal continuity and preventing reliable identity association. To overcome this challenge, we introduce Reliable Identity Memory Tracking (RIMTracking), a joint detection-and-tracking framework that preserves identity consistency by updating trajectory memory only with reliability-screened observations. Specifically, we employ an Observation-Filtered Identity Memory (OFIM) module to recursively consolidate cross-frame observations into a temporally evolving trajectory representation for stable identity association under changing modality reliability. We further develop Scale-Adaptive Cross-Modal Detail Refinement (SCDR), which sparsely samples fine-grained visible and thermal features and injects the aggregated evidence into object queries according to target scale, enhancing small-target localization. Extensive experiments and comparative analyses on VT-Tiny-MOT and VT-MOT demonstrate that our method achieves state-of-the-art performance while consistently improving identity preservation and tiny-object tracking.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
RGB-T multi-object tracking, tiny-object tracking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YDZ8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RIMTracking, an RGB-T multi-object tracking framework designed to improve identity consistency and tiny-object tracking. The framework builds on MOTIP for identity prediction and MS-DETR for multimodal feature extraction. It introduces two components: Observation-Filtered Identity Memory (OFIM), which maintains trajectory-level memory and selectively updates it according to observation and association confidence, and Scale-Adaptive Cross-Modal Detail Refinement (SCDR), which incorporates high-resolution RGB-T features into object queries with scale-dependent weighting.

Critical strengths
  1. Strong empirical performance on relevant RGB-T tracking benchmarks: The proposed framework achieves substantial improvements over the reported baselines on VT-Tiny-MOT and VT-MOT. The gains in HOTA and IDF1 suggest that integrating improved multimodal observations with persistent identity memory can be effective for challenging tracking scenarios

  2. A practically motivated approach to identity memory management: Separating the use of current observations for identity prediction from their incorporation into persistent memory is a sensible design.

Critical weaknesses
  1. The main technical components are closely related to prior work. MeMOTR [1] has already explored EMA-based long-term trajectory memory and memory injection, while SambaMOTR [2] explicitly prevents uncertain observations from contaminating temporal memory through MaskObs. MOTIP [3] provides the ID prediction framework adopted by this paper. Similarly, SCDR combines established ideas from deformable attention, high-resolution feature retrieval, and scale-dependent feature refinement. Although adapting these mechanisms to RGB-T identity prediction may be useful, the paper does not sufficiently demonstrate what fundamental methodological advance is introduced beyond their combination.

  2. The reported improvements from the proposed modules are considerably smaller than the overall gains over earlier trackers. On VT-Tiny-MOT, adding both modules improves HOTA from 34.980 to 35.908. Moreover, the full OFIM write policy improves IDF1 by only 0.055 points over detection filtering with a cooldown interval, suggesting that much of the benefit may arise from a comparatively simple update-frequency heuristic. The component ablation also lacks an OFIM-only configuration, making it difficult to assess the independent contributions and interaction of the two modules.

[1] Gao, Ruopeng, and Limin Wang. "MeMOTR: Long-term memory-augmented transformer for multi-object tracking." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.

[2] Segu, Mattia, et al. "Samba: Synchronized set-of-sequences modeling for multiple object tracking." International Conference on Learning Representations. Vol. 2025. 2025.

[3] Gao, Ruopeng, Ji Qi, and Limin Wang. "Multiple object tracking as id prediction." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

Questions for authors
  1. What is the essential technical difference between OFIM and existing memory mechanisms such as MeMOTR and SambaMOTR's MaskObs? Can the authors provide a controlled comparison with these alternative memory-update strategies using a common tracking backbone?

  2. Can the authors disentangle the contributions of persistent memory reading, unfiltered EMA updating, cooldown-based filtering, and the complete OFIM write policy? In particular, why does detection-plus-identity filtering perform worse than detection-only filtering?

Reviewer_ubjgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses insufficient observations of small objects and identity memory contamination in RGB-T multi-object tracking and proposes RIMTrack. Built upon MS-DETR and MOTIP, RIMTrack employs SCDR to extract local details from high-resolution dual-modal features and adaptively adjusts the enhancement strength according to object scale. OFIM maintains track-level identity memory and selectively updates it after identity assignment based on detection confidence, association confidence, update interval, and scale consistency. Experiments on VT-Tiny-MOT and VT-MOT show that RIMTrack outperforms the compared methods reported in the paper.

Critical strengths
  1. The paper addresses the persistent impact of unreliable observations and incorrect associations on long-term identity memory. Separating frame-level identity association from long-term memory updates provides a clear approach to mitigating error propagation across frames.
  2. The proposed method achieves competitive results on two RGB-T tracking benchmarks.
Critical weaknesses
  1. Although the paper includes module and update-policy ablations, these experiments do not isolate the contribution of memory conditioning or the individual effects of identity-confidence and scale-consistency filtering beyond the detection-plus-cooldown baseline.
  2. In Table 4, introducing identity filtering decreases performance, while the complete OFIM improves HOTA by only 0.005 and IDF1 by 0.055 compared with "detection filtering + update interval."
  3. In Fig. 3, the complete OFIM exhibits a higher proportion of erroneous or unmatched memory updates than detection-only filtering. Since the update acceptance rate decreases from 81.1% to 14.2%, the current results cannot rule out the possibility that the improvements mainly arise from less frequent memory updates.
  4. The main comparison tables omit relevant methods such as MCTrack and ACNTrack, limiting the completeness of the SOTA comparison.
Questions for authors
  1. In Table 4, identity filtering decreases performance, while the improvement of the complete OFIM over "detection filtering + update interval" is marginal. How do the authors explain these results? Do the identity and scale criteria individually provide consistent improvements?
  2. Have the authors considered control experiments with random update selection at a matched update frequency, fixed-interval updates, or a smaller EMA coefficient?
  3. Why are relevant methods such as MCTrack and ACNTrack not included in the main comparison tables?
Reviewer_oLhbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RIMTracking addresses RGB-thermal multi-object tracking of tiny objects, where weak, intermittent detections hurt both localization and identity consistency. It argues that using an observation for the current identity decision should be separated from committing it to long-term identity memory. It builds on MOTIP (association as ID classification) with an MS-DETR RGB-T detector.

Contributions.

SCDR: in decoder layers 4–6, deformable attention retrieves high-resolution (stride-4) RGB and thermal features around each query’s box. A fixed sigmoid gate on box size gives small objects more of this refinement. OFIM: a per-track EMA memory added to the historical ID embeddings before the ID decoder. After assignment, the memory is written only if detection score, top ID probability, ID margin, a 5-frame cooldown and box-area consistency all pass.

Validation.

VT-Tiny-MOT: 35.9 HOTA / 47.6 IDF1, against 29.1 / 43.2 for HGT-Track. VT-MOT: 45.5 HOTA / 56.6 IDF1, against 41.1 / 47.3 for PFTrack. Ablations on VT-Tiny-MOT: SCDR and OFIM added progressively, SCDR also tested with BoT-SORT, five memory-write policies compared, and write-rate and write-purity analyses.

Critical strengths

SCDR helps beyond its own framework. Adding SCDR to MS-DETR with a standard BoT-SORT tracker improves HOTA by 1.5 and IDF1 by 2.6 (Table 3, rows d→e), in addition to its gain inside the ID-decoder pipeline. An unusually transparent analysis of memory writes. Fig. 3 reports write rates and the identity purity of accepted writes for each policy, including a result that doesn’t favor the method (OFIM’s accepted writes are slightly less pure than detection-only filtering).

Critical weaknesses

The gains over prior work come mostly from the base system. Without SCDR or OFIM, the MOTIP + MS-DETR base (row a) already reaches 35.0 HOTA, about 5.9 of the claimed 6.8-point gain over HGT-Track. SCDR then adds 0.5 and OFIM 0.4. The base appears in neither main table, and the large VT-MOT gain (+9.3 IDF1) is credited to OFIM without any ablation on that dataset. As presented, the headline numbers mostly reflect the choice of framework, not the paper’s contributions. OFIM’s evidence points to slower updates, not reliability filtering. Detection + cooldown matches full OFIM (35.903 vs. 35.908 HOTA, Table 4). The identity check alone makes results worse, and OFIM’s accepted writes are no purer than detection-only filtering (28.2% vs. 26.5% wrong or unmatched). What mainly changes is how often memory is written (about 14% vs. 81%). With λ = 0.01, the memory may hardly move from its initial value. All results are single runs reported to three decimals, so the 0.4–0.5 HOTA module effects can’t yet be told apart from seed noise. Since OFIM is the paper’s central idea, this leaves its main claim unsupported.

Questions for authors

How does row (a), MOTIP + MS-DETR, perform on VT-MOT? Can it be added to Tables 1–2? How does OFIM compare with unfiltered EMA at a smaller λ that matches OFIM’s effective update rate, and with an initialize-only memory that is never updated? What is the seed variance for Tables 3 and 4? Does the no-OFIM baseline (row b) get the same third-stage fine-tuning as the full model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.