acceptodds
Under review as a conference paper at ICLR 2027

CDMMT: Conditional Diffusion with Multi-Modal Guidance for Multi-Object Tracking

Abstract

Because data modalities differ substantially, existing multimodal tracking methods often design specialized mechanisms for specific modalities, making it difficult to use multimodal information effectively within a unified framework. To address this issue, we propose Conditional Diffusion for Multimodal Multi-Object Tracking (CDMMT). Through modality-adapted condition encoding and injection, CDMMT incorporates different forms of auxiliary information into a shared paired-box denoising process for object localization and adjacent-frame association. Modality information is first encoded as a condition and injected into paired-box prediction. We further introduce energy-guided sampling: differentiable conditional energies measure the agreement between predicted boxes and modality information, and their gradients correct intermediate box predictions so that the constraints propagate to later predictions through sampling-state updates. Using Global Navigation Satellite System (GNSS) positions and infrared images, we demonstrate the encoding, injection, and guidance of sparse positional and dense image information. Experiments show that CDMMT effectively uses multiple modalities and improves tracking performance. Code is available anonymously at https://anonymous.4open.science/r/CDT-B163.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal tracking, multi-object tracking, conditional diffusion model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pvSxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CDMMT, a conditional diffusion framework for multimodal multi-object tracking. Building upon paired-box diffusion, the framework incorporates auxiliary modalities through modality-specific condition encoding and injection. It further introduces energy-guided sampling, where gradients of differentiable conditional energies are used to correct intermediate box predictions during denoising. The authors demonstrate the approach using simulated GNSS positioning and infrared images, and report tracking results on AGVS-T24 and VT-Person-MOT.

Critical strengths
  1. The paper considers two distinct forms of auxiliary input: sparse object-level positions and dense thermal images. Incorporating these different information types into a shared paired-box prediction framework is a reasonable design that may benefit tracking systems operating with heterogeneous sensors.

  2. The reported results demonstrate substantial improvements in several tracking metrics, particularly MOTA and DetA in infrared-guided tracking. The valid-versus-invalid condition experiments also provide evidence that meaningful auxiliary information, rather than merely adding conditional branches, contributes to the reported performance.

Critical weaknesses
  1. The proposed framework largely builds upon DiffusionTrack [1], which already formulates multi-object detection and association as a paired-box diffusion process. CDMMT retains this core formulation and mainly extends it by introducing modality-specific condition injection and energy-guided sampling. However, both conditional feature injection and gradient-based guidance during diffusion sampling have been extensively explored in prior works, such as ControlNet, Uni-ControlNet, and Universal Guidance. The proposed method therefore appears to primarily adapt and combine existing techniques within an established tracking framework, rather than introducing a fundamentally new diffusion or association mechanism. Moreover, related work such as DM³T [2] has already explored the integration of diffusion models with multimodal multi-object tracking. The paper does not sufficiently distinguish its contributions from these closely related approaches, making the overall methodological novelty appear incremental and limited.

  2. The experiments lack key ablations separating condition injection from energy-guided sampling, making it difficult to assess the contribution of the proposed guidance mechanism. Furthermore, the GNSS conditions are synthesized from ground-truth annotations, raising concerns about evaluation fairness.

[1] Luo, Run, et al. "Diffusiontrack: Diffusion model for multi-object tracking." Proceedings of the AAAI conference on artificial intelligence. Vol. 38. No. 5. 2024.

[2] Li, Weiran, et al. "DM^ 3T: Harmonizing Modalities via Diffusion for Multi-Object Tracking." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

Questions for authors

What are the precise technical innovations beyond DiffusionTrack and existing diffusion guidance methods? In particular, how does the proposed energy-guided update fundamentally differ from applying a conventional external guidance gradient to predicted clean box coordinates?

Reviewer_MVRhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1(−1)
Soundness
2
Read review
Paper summary

The paper proposes CDMMT, a multimodal multi-object tracking framework that predicts corresponding bounding boxes in adjacent frames through conditional denoising. It incorporates sparse GNSS position inputs through cross-attention and dense infrared features through multiscale residual fusion, while retaining a shared paired-box prediction framework. It also proposes inference-time energy guidance to adjust intermediate box predictions using positional agreement or cross-frame infrared feature consistency. A trajectory-management stage maintains identities across the video.

Evaluation covers AGVS-T24 with simulated GNSS positions and identities derived from ground-truth annotations, a visible–thermal pedestrian subset of VT-MOT, and a supplementary depth-conditioned extension on MOT17. The paper reports improvements over RGB-only DiffusionTrack and provides condition-quality ablations, auxiliary initialization and association ablations, and repeated GNSS inference runs. These experiments support the usefulness of auxiliary information, although the reported single-step sampling configuration does not seem to activate the proposed energy-guidance mechanism.

Critical strengths
  1. A coherent framework for heterogeneous auxiliary inputs. The method accommodates sparse positioning data and dense infrared images within the same paired-box tracking formulation, using modality-specific encoding and fusion. This is a useful design choice because these inputs have substantially different structures, yet can support a common localization and association pipeline.

  2. Meaningful gains from real infrared information. On VT-Person-MOT, the method improves HOTA from 25.6 to 41.8 and IDF1 from 27.7 to 48.5 over RGB-only DiffusionTrack. The condition-quality ablation further shows that valid infrared images substantially outperform both absent and all-zero conditions. This provides concrete evidence that the model uses informative thermal cues, rather than benefiting merely from the presence of an additional branch, supporting its practical value when RGB observations are insufficient.

Critical weaknesses
  1. The reported experiments do not validate the proposed energy-guidance mechanism. Supplementary Table 3 specifies one DDIM sampling step for both GNSS and infrared experiments, while Algorithm 1 applies energy correction only before a subsequent sampling step. The supplementary text explicitly states that no energy step is performed when K=1K=1. Thus, under the documented configuration, a prominently presented methodological component is inactive. Without clarification and a controlled comparison with and without energy guidance at the same sampling budget, its contribution remains unsupported.

  2. The comparisons do not sufficiently separate methodological gains from access to additional information. The paper explicitly identifies DiffusionTrack as RGB-only, but does not clearly document whether the other baselines receive equivalent auxiliary inputs or provide clearly specified competitive multimodal baselines. This is particularly consequential for GNSS: simulated positions and identities are derived from ground-truth annotations, and the full system uses those identities for auxiliary association. While this is a disclosed simulation setting, comparisons against methods without equivalent information cannot establish the advantage of the proposed design. The condition-quality ablations demonstrate that auxiliary information is useful, but not that CDMMT exploits it better than simpler alternatives receiving the same inputs.

Questions for authors
  1. As stated in the weaknesses, Supplementary Table 3 specifies one DDIM sampling step for both modalities, whereas Algorithm 1 applies energy guidance only when another sampling step follows. The text also states that K=1K=1 performs no energy correction. Was energy guidance enabled in any reported experiment? If not, then the reported results cannot support the claimed contribution of energy guidance, as they do not evaluate this component.

  2. For each baseline in Table 1, please specify whether it receives RGB only or the same auxiliary information available to CDMMT, including GNSS positions and identities or infrared images. Why are relevant multimodal MOT baselines, particularly RGB–thermal trackers, not included? Are there controlled comparisons against simpler fusion or association methods using identical auxiliary inputs? The condition-quality ablations demonstrate the usefulness of additional information, but do not establish whether the proposed design uses that information more effectively than simpler alternatives. Without such comparisons, it is difficult to attribute the reported gains to the proposed method rather than access to additional information.

  3. The GNSS positions and IDs are generated from ground-truth annotations. During testing, does every target receive a GNSS position and a correct, consistent ID in every frame? How much position error is added, and how is that error generated? Are any GNSS observations missing, delayed, or associated with the wrong target? Please clarify the noise added to the GNSS observations themselves, rather than the noise used to initialize candidate boxes. These details are important for understanding whether the reported improvements would also hold with real GNSS inputs.

Reviewer_rbvJOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

CDMMT proposes a unified conditional diffusion framework for multi-object tracking that integrates heterogeneous modalities — GNSS positioning (sparse 2D coordinates) and infrared imagery (dense images) — into a single paired-box denoising process.

Critical strengths

Two condition channels with matched granularity — sparse token-level (GNSS) vs. dense residual-level (IR) — share a single estimator and sampling rule; the architecture is clean and naturally extensible to radar or depth modalities.

Critical weaknesses

GNSS conditioning data is constructed from ground-truth annotations. §4.1 states: "As synchronized GNSS data are unavailable, we construct perturbed two-dimensional positions from object annotations." Because E_loc (Eq. 15) pulls each predicted box toward the nearest valid position — a position derived from GT — the GNSS setting (Tab. 1/2) approximates test-time supervision with near-GT coordinates. The perturbation protocol (noise distribution, amplitude, temporal consistency, projection model) is never disclosed. Claims that GNSS guidance is effective are therefore unsupported in the current form.

Questions for authors

1.Can you provide results with real GNSS trajectories? If such data is unavailable, please disclose the full perturbation protocol (distribution, amplitude, temporal consistency, projection error) and restate the claim as "conditioning under approximate localization."

2.Why DDIM is used for these modality related fustion problem?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.